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From:  Director,  Marine  Corps  Operational  Test  and  Evaluation  Activity 
To:  MCOTEA  All  Hands 

Sub j  :  MCOTEA  OPERATIONAL  TEST  &  EVALUATION  MANUAL  SECOND  EDITION 

1.  The  MCOTEA  Operational  Test  and  Evaluation  Manual  presents  a 
process  rooted  in  both  the  scientific  method  and  Marine  Corps 
operations.  The  manual  combines  elements  of  Marine  Corps  missions  and 
tasks  with  systems  engineering,  decision  analysis,  and  design  of 
experiments  to  provide  a  process  that  supports  all  test  and  evaluation 
activities  that  MCOTEA  performs. 

2.  This  second  edition  of  the  manual,  which  supersedes  version  1.1, 
has  been  updated  with  significant  new  material  on  Reliability, 
Availability,  and  Maintainability  as  well  as  an  in-depth  section  that 
sets  forth,  for  the  first  time,  a  MCOTEA  process  for  accrediting 
models  and  simulation.  The  overall  test  and  evaluation  process  as 
established  in  version  1.1  has  not  changed  apart  from  adjustments  and 
clarifications  based  on  lessons  learned.  The  organization  of  the 
manual  has  been  improved  to  better  reflect  and  follow  MCOTEA' s  six- 
step  test  and  evaluation  process. 

3.  This  manual  is  a  living  document  and  will  be  updated  regularly  with 
additional  material.  All  hands  are  encouraged  to  submit  comments  or 
recommendations  to  the  Scientific  Advisor  for  the  improvement  of  this 
manual . 

4 .  Use  of  this  manual  for  performing  Marine  Corps  operational  test  and 
evaluation  is  mandatory  and  effective  immediately. 

Thank  you  for  all  your  continued  professionalism  and  cooperation. 


D.L.  REEVES 


MCOTEA  Mission 

MCOTEA  provides  operational  testing  and  evaluation 
for  the  Marine  Corps  and  conducts  additional  testing  and 
evaluation  as  required  to  support  the  Marine  Corps  mission 
to  man,  train,  equip,  and  sustain  a  force  in  readiness. 


MCOTEA  Vision 

MCOTEA  will  be  the  Marine  Corps  leader  in  all  aspects  of 
realistic  operational  test  and  evaluation  of  materiel  system 
capabilities  throughout  a  materiel  system’s  life  cycle.  Our 
highly  trained,  professional  workforce  will  be  a  voice  for 
the  Operating  Force  Marine,  enabling  informed  decision¬ 
making,  and  ensuring  always  that  our  test  reports  accurately 
and  objectively  describe  what  we  know  and  don’t  know  about 
the  Operational  Effectiveness,  Suitability,  and  Survivability 
of  the  materiel  solution  we  evaluate.  MCOTEA  will  be 
a  source  for  objectivity  in  the  Marine  Corps  and,  where 
appropriate,  DOD’s  acquisition  process.  Our  expertise, 
professionalism,  and  integrity  will  make  us  a  sought-after 
partner  within  the  DOD  acquisition  community. 
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Organization 


General  Philosophy 

MCOTEA  is  organized  into  an  Executive 
Office,  Test  Divisions,  and  Staff  sections  (figure 
1-1). These  components  support  the  Director 
in  accomplishing  all  of  the  functions  assigned 
to  MCOTEA  to  ensure  realistic,  rigorous, 
independent,  and  unbiased  Operational  Test 
and  Evaluation  (OT&E)  for  the  Marine  Corps. 

This  section  briefly  describes  each 
component  of  the  MCOTEA  organization. 
These  descriptions  are  an  overview  and  do 
not  attempt  to  cover  all  of  the  functions 
associated  with  each  component. 

Executive  Office 

Director 

The  Director,  MCOTEA,  with  support 
from  the  Test  Divisions  and  staff,  ensures 
the  effective  performance  of  all  the  top- 
level  functions  discussed  in  chapter  2  and 
the  following  additional  responsibilities 
(Secretary  of  the  Navy  2008): 

♦  Host  and  chair  a  Test  and  Evaluation  (T&E) 
Working-level  Integrated  Product  Team 
(WIPT)  to  determine  Failure  Definition/ 
Scoring  Criteria  (FD/SC)  for  each  program 

♦  Request  the  assignment  of  Test  Director  for 
Aquisition  Category  (ACAT)  I  and  certain 
ACAT  II  programs  from  the  Assistant 
Commandant  of  the  Marine  Corps 
(ACMC), 

♦  Advise  the  Milestone  Decision  Authority 
(MDA)  of  the  associated  risks  in  the 
procurement  decision  when  significant  test 
limitations  are  identified 

♦  Conduct  an  Operational  Test  Readiness 
Board  (OTRB)  to  determine  MCOTEA’s 
readiness  to  proceed  with  OT&E 

♦  Advise  the  ACMC  on  all  OT&E  matters 

♦  Chair  an  annual  OT&E  planning 
conference  with  representation  from  the 
Marine  Operating  Forces;  appropriate 


HQMC  staff  offices;  DC,  CD&I;  CG, 
MCSC;  and  others  as  appropriate. 

♦  Maintain  direct  liaison  with  the  Office  of 
the  Secretary  of  Defense  (OSD)  Director, 
Operational  Test  and  Evaluation  (DOT&E), 
the  Marine  Operating  Forces  for  OT&E 
matters,  other  DOD  agencies,  military 
activities,  and  commands  as  required 

♦  Concur  with  the  LFT&E  strategy  as  planned 
in  the  Test  and  Evaluation  Strategy  or  Test 
and  Evaluation  Master  Plan  (TEMP),  and  as 
approved  by  the  MDA  for  USMC  programs 
not  required  by  statute  to  conduct  Live  Fire 
Test  and  Evaluation  (LFT&E),  but  where 
LFT&E  is  appropriate. 

Deputy  Director 

The  Deputy  Director,  MCOTEA 
assists  the  Director  in  performing  his 
responsibilities  and  directs  the  efforts 
of  the  staff  in  supporting  the  Director 
and  executing  MCOTEA  functions. 

The  Deputy  supports  the  Director  in 
determining  the  future  direction  of  and 
vision  for  MCOTEA.  The  Deputy  also 
represents  MCOTEA  in  various  forms  by 
interfacing  with  external  organizations. 

Scientific  Advisor 

The  Scientific  Advisor  (SA)  provides 
technical  advice  on  evaluation  strategies, 
test  planning,  and  test  execution  and 
provides  quality  assurance  for  MCOTEA 
products.  The  SA  tracks  DOD  and 
Department  of  the  Navy  (DON)  policies 
and  interprets  how  they  affect  MCOTEA. 
In  addition,  the  SA  assists  the  Director  and 
the  Deputy  in  determining  MCOTEA’s 
future  direction.  The  SA  investigates  new 
testing  and  evaluation  methodologies 
and  instrumentation  that  are  applicable 
to  MCOTEA.  The  SA  also  interfaces 
with  external  organizations,  representing 
MCOTEA  in  various  forums. 
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Finally,  the  SA  leads  the  MCOTEA 
efforts  in  process  improvement.  In  this  role, 
the  SA  obtains  input  from  MCOTEA 
staff  members  and  recommends  process 
improvements  and  changes  to  the  Director. 

Chief  of  Staff 

The  Chief  of  Staff  (COS)  serves  as  the 
overall  staff  lead  under  the  cognizance  of 
the  Deputy  Director.  The  COS  ensures 
that  the  staff  executes  the  Director’s 
guidance  in  a  coordinated  and  integrated 
manner.  The  COS  also  ensures  timely, 
efficient,  and  effective  coordination  of  staff 
efforts  in  support  of  the  test  divisions.  The 
COS  is  responsible  for  implementing  the 
MCOTEA  Safety  Program. 


Figure  1-1. 

Test  Divisions  MCOTEA's  Organization 

Testing  is  accomplished  in  the  four  Test 
Divisions,  each  of  which  comprises  three 
branches.  The  Test  Divisions  ensure  that 
sufficient  and  qualified  personnel  are 
assigned  to  each  test  program.  The  divisions 
also  ensure  that  MCOTEA  testing  is 
well  planned,  well  coordinated,  and  has 
sufficient  materiel  support.  In  addition, 
the  Divisions  generate  the  final  documents 
that  report  on  accomplished  testing.  Each 
division  is  run  by  a  Division  Head  who 
acts  as  the  Assistant  Contracting  Officer 
Representative  (ACOR)  for  all  program 
tasks  within  their  respective  divisions. 

The  Divisions  provide  services  to  the 
Marine  Corps,  Multi-Service,  and  Joint 
Service  organizations  and  perform  various 
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levels  of  testing  depending  on  system 
complexities  and  the  decision  maker’s 
needs.  The  Test  Divisions  work  in  close 
coordination  with  the  lead  Operational 
Test  Agency  (OTA)  for  programs  requiring 
Multi-Service  Operational  Test  and 
Evaluation  (MOT&E). 

Combat  Service  Support  Test  Division 

Combat  Service  Support  Test  Division 
(CSSTD)  is  responsible  for  monitoring 
and  testing  programs  associated  with 
individual  items  for  personnel  combat 
survivability  and  motor  transport  assets 
(Combat  Service  Support  Test  Branch); 
combat  engineering  equipment  (Combat 
Engineer  Test  Branch);  and  Chemical, 
Biological,  Radiological,  and  Nuclear 
(CBRN)  equipment  related  to  chemical 
and  biological  detection  and  protection 
efforts  (CBRN  Test  Branch). 

Expeditionary  Test  Division 

Expeditionary  Test  Division  (ETD)  is 
responsible  for  monitoring  and  testing 
programs  associated  with  USMC 
amphibious  vehicles  (Amphibious 
Vehicle  Branch);  Navy  ship  and  ship- 
to-shore  connector  programs  (Naval  Test 
Branch);  and  supports  MCOTEA  forward 
operations  (FOA  Branch). 

Ground  Combat  Test  Division 

Ground  Combat  Test  Divion  (GCTD) 
is  responsible  for  monitoring  and  testing 
programs  associated  with  infantry  weapon 
systems  and  infantry  combat  equipment 
(Infantry  Test  Branch);  artillery  and  artillery 
support  equipment  (Fires  Test  Branch);  and 
combat  vehicle,  anti-armor,  non-lethal,  and 
robotics  (Combat  Vehicles  Test  Branch). 

MAGTF  C4ISR 

The  Marine  Air-Ground  Task  Force 
(MAGTF)  Command,  Control, 
Communications,  Computers, 

Intelligence,  and  Reconnaissance  Test 
Division  (C4ISRTD)  is  responsible 


for  monitoring  and  testing  programs 
associated  with  Marine  Corps 
information,  command,  control,  and 
intelligence  systems  (C4ISRTest 
Branch);  command  and  control  systems 
(MAGTF  Command  and  Control  (C2) 
Test  Branch);  information  systems, 
communications  and  networking  systems, 
simulators,  and  the  Information  Assurance 
(IA)  Range  (Information  Systems  Test  Branch). 

Staff  Functions 

Staff  Sections  support  the  Director  in 
executing  all  MCOTEA  functions.  In 
particular,  the  staff  supports  the  Test 
Divisions  by  ensuring  that  testing  and 
evaluation  is  well  planned  and  coordinated, 
adequately  staffed,  and  has  sufficient 
materiel  support.  The  staff  also  helps  ensure 
that  internal  processes  are 

♦  efficient  and  consistent  with  higher-level 
directives 

♦  contribute  to  the  delivery  of  high-quality 
products 

♦  support  effective  communication  and 
coordination  with  external  agencies 

Staff  Section  numbering  and  functions 
reflect  common  MAGTF  usage  where 
possible  to  facilitate  communication  with 
Marine  Corps  organizations.  Each  Staff 
Section  is  run  by  a  Staff  Fead,  who  acts  as 
the  ACOR  for  all  program  tasks  within  the 
respective  Staff  Sections. 

Chief  of  Test 

The  Chief  ofTest  (COT)  is  responsible  for 

♦  developing,  auditing,  improving,  and 
enforcing  MCOTEA  processes 

♦  providing  guidance  for  evaluation  strategies,  test 
planning,  execution,  analysis,  and  reporting 

♦  providing  quality  control  and  quality 
assurance  of  MCOTEA  products 

♦  processing  Warfighter  feedback  from  the  fleet 

♦  approving  Accreditation  plans  and  reports 

In  coordination  with  the  S-2  and  the  SA, 
the  COT  ensures  that  MCOTEA  employs 
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the  most  efficient  and  effective  test 
methodologies  and  instrumentation. 

Lead  Contract  Integrator 

The  Contract  Support  team  establishes  and 
manages  contract  support  for  MCOTEA 
to  include  the  Test  Divisions  and  Staff  as 
well  as  meeting  other  MCOTEA-level 
requirements.  The  support  team  consists 
of  a  Lead  Contract  Integrator  (LCI), 
Contract  Specialist,  and  Administrative 
Support  Specialist  and  combines  efforts 
with  outside  agency  support  members 
providing  contract  warrant  authority. 
External  agency  support  typically  includes 
a  Procuring  Contract  Officer,  Contract 
Specialist,  and  Contract  Intern. 

The  Contract  Support  Team 

♦  develops  and  executes  comprehensive 
contracting  support  strategies  (e.g.,TS- 
SCI  requirements,  Operational  Conflict  of 
Interest  avoidance,  and  sustained  support) 

♦  coordinates  with  Marine  Corps  and  other 
agencies  inside  and  outside  DOD  in 
support  of  contract  services  requirements 

♦  ensures  that  pre-award  and  post-award 
contracting  activities  are  carried  out  in 
accordance  with  policies  and  regulations 

♦  establishes  and  manages  internal  contracting 
processes  (quality  assurance  and  quality 
control)  to  ensure  efficient  and  effective 
support 

♦  provides  oversight  of,  or  conducts  actual 
execution  of,  contract  activities  to  include 
contracting  officer  representative  activities 

♦  coordinates  development  of  cost  estimates 
and  independent  Government  cost  estimates 
supporting  the  establishment  of  contracts. 

S-1:  Human  Capital  and 
Administration 

The  S-1  is  the  primary  advisor  to  the 
Director,  Deputy  Director,  Divisions,  and 
Staff  on  all  military  and  civilian  personnel 
matters  and  maintains  accountability  of  all 
personnel.  The  S-1  is  responsible  for 


♦  developing  plans,  policies,  procedures,  and 
programs  related  to  military  and  civilian 
human  capital  administration 

♦  overseeing  the  Unit  Table  of  Organization 

♦  recommending  manpower  allocation  in 
collaboration  with  the  Staff  Leads  and 
Division  Heads 

♦  coordinating  training  for  MCOTEA 
military  and  civilian  personnel 

♦  performing  administrative  support  functions 
including  awards,  correspondence,  archiving, 
personnel  evaluations,  mail,  security, 
Freedom  of  Information  Act  requests,  travel 
authorization,  etc. 

♦  maintaining  the  MCOTEA  Test  and 
Evaluation  Reference  Center  (see  chapter  5) 

S-2:  Decision  Sciences 

The  S-2  provides  decision  science 
capabilities  in  evaluation  strategy,  analytical 
test  design,  and  test  concept  development. 
The  S-2  is  also  responsible  for  providing 
specialty  services  including 

♦  IA  assessment 

♦  use  of  Modeling  &  Simulation  (M&S) 

♦  accrediting  M&S  for  MCOTEA  use 

♦  assessing  Live  Fire  and  Survivability 

♦  developing  techniques  for  determining 
Reliability,  Availability,  and  Maintainability 
(RAM) 

♦  use  of  Human  Factors  in  test  planning  and 
system  evaluation 

All  new  efforts  enter  MCOTEA  through 
the  S-2,  where  an  initial  evaluation  strategy 
is  formed  and  presented  to  the  Test 
Divisions.  The  S-2  stays  informed  about 
new  evaluation  and  test  methodologies 
and  instrumentation  and  proposes  their 
application  to  testing  and  evaluation. 

S-3:  Operations 

The  S-3  coordinates  and  manages 
MCOTEA  organizational  tasks  related 
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to  external  agencies.  The  S-3  coordinates 
MCOTEAs  attendance  and  participation  at 
the  Force  Synchronization  and  Coordination 
Conferences  and  coordinates  Commanders’ 
Conferences,  ceremonies,  change  of  command, 
and  other  events.  The  S-3  supports  the 
Divisions  by  coordinating  MCOTEAs 
test  schedules,  test  range  usage,  and  Digital 
Message  Service  message  traffic.  The  S-3  also 
handles  protocol  issues  and  public  affairs. 

S-4:  Logistics 

The  S-4  is  responsible  for 

♦  managing  all  Government-furnished 
equipment  at  the  MCOTEA  facility  and 
test  sites 

♦  coordinating  the  transportation  of  personnel 
and  equipment  to  test  sites 

♦  fully  supporting  test  site  logistics 

♦  managing  Information  Technology  assets 
including  NMCI,  VTC,  classified  networks, 
telephone  and  BlackBerry®  services,  and 
help  desk  functions 

♦  maintaining  an  accurate  and  up-to-date 
inventory  of  MCOTEA’s  resources 

S-5:  Future  Operations 

The  S-5  seeks  out  strategic  initiatives  for 
MCOTEA,  enabling  the  use  of  MCOTEAs 
expertise  in  a  broad  range  of  programs.  In 
addition,  the  S-5  provides  MCOTEA  with 
long-range  assessments  of  emerging  T&E 
trends  and  requirements.  Furthermore,  the 
S-5  looks  for  gaps  in  USMC  manpower, 
equipment,  and  training,  and  recommends 
ways  in  which  MCOTEA  can  help  the 
Marine  Corps  address  these  gaps. 


Fiscal 

The  Fiscal  Office  manages  all  funds 
received  throughout  the  year  for 
Operations  and  Maintenance  Marine 
Corps  (O&MMC);  Research, 
Development,  Test,  and  Evaluation 
(RDT&E);  and  other  programs.  Fiscal  also 
develops  Program  Objective  Memorandum 
(POM)  briefs  for  consideration  in  the 
overall  RDT&E  and  O&MMC  POM 
submissions,  and  submits  POM  and  budget 
exhibits  justifying  the  request  for  resources. 

In  addition,  Fiscal 

♦  manages  and  monitors  transaction  source 
documents 

♦  oversees  the  development  of  civilian  labor 
cost  projections 

♦  approves  all  credit  card  purchases  and 
training  requests 

♦  manages  Procurement  Request  builder 

♦  oversees  the  Defense  Travel  System  program 

♦  accepts  invoices  in  Wide  Area  Work  Flow 

The  Test  Team 

See  chapter  2  for  detailed  information 
about  the  test  team. 
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Integrated  testing  is"the 
collaborative  planning  and 
collaborative  execution 
of  test  phases  and  events 
to  provide  shared  data  in 
support  of  independent 
analysis,  evaluation, 
and  reporting  by  all 
stakeholders,  particularly 
the  developmental  (both 
contractor  and  government) 
and  operational  test  and 
evaluation  communities" 
(Office  of  the  Secretary  of 
Defense  2008). 


Background  &  Paradigm 


MCOTEA’s  Mandate 
and  Purpose 

MCOTEA  is  the  independent  OTA  for 
the  United  States  Marine  Corps  (SECNAV 
2008).  In  this  capacity,  MCOTEA  provides 
information  to  the  MDA  as  part  of  the 
decision-making  process  for  acquiring 
solutions  that  satisfy  validated  user  needs. 
MCOTEA  serves  the  MDA,  the  USMC, 
and  the  DOD  by  objectively  evaluating, 
under  operational  conditions,  how  well  a 
solution  meets  required  mission  capabilities. 
MCOTEA’s  role  is  to  ensure  that  deployed 
systems  accomplish  their  missions 
effectively  without  imposing  unreasonable 
requirements  on  field  support  infrastructure. 

The  fundamental  purpose  of  Initial 
Operational  Test  and  Evaluation 
(IOT&E)  is  to  assist  in  managing  the 
risks  involved  in  developing,  producing, 
fielding,  operating,  and  sustaining  systems 
and  capabilities.  Initial  Operational  Test 
(IOT),  preceded  by  the  materiel  developer’s 
developmental  testing,  investigates  the 
Operational  Effectiveness,  Operational 
Suitability,  and  Operational  Survivability 
(OE/OS/OSur)  of  an  acquisition  system. 
MCOTEA  assists  program  acquisition  by 
collaboratively  planning  and  participating 
in  integrated  test  events,  observing 
developmental  test  events,  and  providing 
Observation  and  Assessment  Reports 
throughout  the  acquisition  cycle. 

Evaluation  of  test  data  from  integrated 
testing  and  IOT  provides  a  basis  for 
assessing  system  performance.  System 
evaluation  is  typically  an  overarching 
strategy  that  gathers  information  from 
multiple  developmental  and  operational 
test  events. 

MCOTEA  strives  to  provide  decision 
makers  with  timely  information  on  program 
capabilities  and  limitations.  To  accomplish 
this,  MCOTEA  ensures  that  each  system 
proposed  for  acquisition  is  tested  adequately, 


evaluated  objectively,  and  reported  on 
independently.  Integrated  testing  and  system 
evaluation  allow  the  acquisition  community 
to  learn  about  and  correct  or  mitigate  a 
system’s  operational  limitations  before  full- 
rate  production  (FRP)  and  deployment.  In 
turn,  a  fielded  system’s  user  community  can 
apply  knowledge  gained  from  IOT&E  to 
optimize  system  use. 

To  properly  measure  a  system’s  capabilities, 
MCOTEA  uses  a  Mission-Based  Testing 
approach  and  custom  designs  each 
evaluation  strategy.  Test  planning  focuses 
on  the  missions  the  system  is  designed 
to  support.  Top-level  requirements  for 
adequate  operational  testing  are  as  follows: 

♦  employ  a  production-representative  system 
in  realistic  operating  conditions  with  typical 
Marine  operators  and  maintainers 

♦  collect  data  that  accurately  describes  the  test 
conditions  and  system  performance  results 

♦  analyze  the  data  independently  and  without 
bias  for  use  in  system  evaluation 

Top-level  requirements  for  objective  system 
evaluation  are  as  follows: 

♦  collect  and  evaluate  information  from  a 
variety  of  developmental  and  operational 
test  events 

♦  determine  if  thresholds  in  the  approved 
capabilities  documentation  and  Critical 
Operational  Issues  (COI)  have  been  satisfied 

♦  determine  the  system’s  OE/OS/OSur 

♦  assess  system  effects  on  combat  operations 

♦  provide  any  additional  information  on  the 
system’s  operational  capabilities 

OE/OS/OSur 

■=>  0E  is  based  on  mission  success 

■=>  OS  is  based  on  factors  that  affect  mission 
accomplishment 

■=>  OSur  is  based  on  the  degree  to  which  the  system 
outs  operators  at  risk 
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MCOTEA’s  Working 
Relationships  with  Other 
Organizations 

MCOTEA  reports  directly  to  the  ACMC 
and  interacts  with  other  organizations  at 
various  levels  and  to  varying  degrees  (fig.  2-1). 

Working  Partners 

MCOTEA’s  closest  working  partners, 
Deputy  Commandant  for  Combat 
Development  and  Integration  (DC, 

CD&I)  and  Marine  Corps  Systems 
Command  (MCSC)/Program  Executive 
Officer  Land  Systems  (PEO  LS),  form 
the  acquisition  “triad”  with  MCOTEA. 
MCOTEA  staff  works  most  closely  with 
these  entities. 

Deputy  Commandant  for  Combat 
Development  and  Integration 

The  DC,  CD&I  is  responsible  for 
identifying  gaps  in  combat  capabilities 
and  for  generating  the  Joint  Capabilities 
Integration  Development  System 
(JCIDS)  documents  to  address  these 
gaps,  including  the 

♦  Initial  Capabilities  Document  (ICD) 

♦  Capability  Development  Document  (CDD) 

♦  Capability  Production  Document  (CPD) 

♦  Concept  of  Operations  (CONOPS) 

♦  Concept  of  Employment  (COE) 

MCOTEA  works  closely  with  the 
DC,  CD&I  organization,  primarily  the 
Capabilities  Development  Directorate, 
very  early  in  the  system’s  acquisition  cycle 
to  help  ensure  that  requirements  are 
testable  and  that  MCOTEA  understands 
the  context  in  which  the  requirements 
were  generated.  For  the  purposes  of  this 
manual,  the  term  DC,  CD&I  is  used  to 
describe  the  capabilities  development 
functions  of  DC,  CD&I. 


Marine  Corps  Systems  Command 


MCSC  is  the  Commandant’s  agent 
for  acquiring  and  sustaining  systems 
and  equipment  used  to  accomplish  the 
warfighting  mission.  MCSC  addresses 
system  capabilities  and  requirements 
generated  by  DC,  CD&I.  MCOTEA 
works  closely  with  MCSC  from  early  in 
the  acquisition  cycle  to  after  IOT  to  help 
mitigate  program  risk.  The  Commander, 
MCSC  is  the  Marine  Corps  Executive 
Agent  for  DT. 


Fellow  OTAs 


ATEC 

COTF 

AFOTEC 

JITC 


Committee  membership/ 
info  exchange 

T&E  BOD 
N09I 
OTICC 
TRMC 


Program  Executive  Officer  Land  Systems 

PEO  LS  partners  with  MCSC  to  develop, 
deliver,  and  provide  lifecycle  planning 
for  assigned  programs.  As  with  MCSC, 
MCOTEA  works  closely  with  PEO  LS 
from  early  in  the  acquisition  cycle  to  after 
IOT  to  help  mitigate  program  risk.  As  with 
MCSC,  MCOTEA  observes  developmental 
testing  and  conducts  assessments  on  systems 
with  PEO  LS  and  conducts  IOT  on  selected 
systems  as  required. 

Oversight/Non-Chain  of  Command 

Director,  Operational  Test  and  Evaluation 

The  DOT&E  in  the  OSD  is  the  principal 
OT&E  official  within  the  DOD. 

DOT&E’s  job  is  to  help  ensure  that  a 
system  is  operationally  effective  and  suitable 
before  going  beyond  Low-Rate  Initial 


Oversight  and  Directives 
MCOTEA  must  follow 

DOT&E 
ASN  (RDA) 


Working  Partners 

MCSC 
DC,  CD&I 
PEO  LS 


Figure  2-1. 

MCOTEA's  Relationship  with 
Other  Organizations 
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Production  (LRIP).  Stated  another  way, 
DOT&E’s  primary  interest  is  to  ensure  that 
OT&E  and  LFT&E  are  adequate  before 
FRP  or  deployment,  and  that  tests  and 
evaluations  are  properly  executed  according 
to  statute  and  DOD  policy. 

Although  not  in  the  MCOTEA  chain 
of  command,  DOT&E  has  significant 
oversight  over  any  MCOTEA  programs 
on  the  DOD  T&E  Oversight  List,  and 
MCOTEA  is  required  to  conform  to 
DOT&E  guidance  for  these  programs. 

Any  program,  regardless  of  Acquisition 
Category  level,  can  be  included  on  the 
T&E  Oversight  List.  Selection  criteria 
include  ACAT  level,  Congressional  and/ 
or  DOD  interest,  programmatic  risk  level, 
technical  complexity,  and  relationship  with 
other  systems.  All  “oversight”  programs 
require  additional  briefings,  reports,  and 
supporting  documentation  and  often 
require  additional  testing.  The  DOT&E 
web  site  at  http://www.dote.osd.mil 
contains  the  Annual  T&E  Oversight 
List.  The  Defense  Acquisition  Guidebook 
(DAU  2009)  contains  additional  details. 

DOT&E’s  primary  responsibility  for 
Oversight  List  programs  is  to  provide  final 
approval  for  the  TEMP  before  milestone 
decision  reviews  and  to  approve  OT&E 
plans  before  those  tests  may  commence. 

No  operational  testing  may  occur  for 
a  program  on  the  Oversight  List  until 
DOT&E  has  provided  written  approval 
of  the  OT&E  plans.  Early  involvement  of 
DOT&E  personnel  in  drafting  the  T&E 
strategy,  the  TEMP,  and  operational  test 
plans  for  programs  on  the  Oversight  List 
will  help  ensure  smooth  approval. 

Assistant  Secretary  of  the  Navy 
(Research,  Development,  and 
Acquisition) 

Assistant  Secretary  of  the  Navy  (Research, 
Development,  and  Acquisition)  (ASN 
(RDA))  is  the  DOD’s  Component 
Acquisition  Executive  for  acquisition 
activity,  including  test  and  evaluation.  ASN 


(RDA)  provides  DON-level  acquisition 
and  T&E  guidance  to  supplement 
guidance  from  DOD.  Although  not 
in  the  MCOTEA  chain  of  command, 
MCOTEA  is  required  to  conform  to  ASN 
(RDA)  T&E  guidance. 

DON  uses  the  Gate  Review  process  to  help 
monitor  programs  of  interest.  The  Gate 
Review  process  provides  a  framework  for 
engaging  senior  naval  leadership  on  certain 
acquisition  programs  to  improve  decision 
making  through  better  understanding  of 
program  risks  and  costs  (SECNAV  2008). 

Gate  Reviews 

The  Gate  Review  process  helps  ensure 
alignment  between  capability  requirements 
and  acquisition  while  improving  senior 
leadership  visibility  into  program  risks 
and  costs  throughout  the  development 
cycle.  DON  has  adopted  the  Probability  of 
Program  Success  (PoPS)  approach,  used  in 
conjunction  with  Gate  Reviews,  to  assess 
and  monitor  the  health  of  naval  acquisition 
programs.  Program  health  is  subdivided 
into  17  metrics,  one  of  which  is  T&E. 

Six  Gate  Reviews  are  distributed  over 
two  “passes.”  Figure  2-2  shows  where 
the  Gate  Reviews  fall  in  the  acquisition 
process.  The  first  three  gates  constitute  the 
“requirements”  gates  while  the  last  three 
constitute  the  “acquisition”  gates.  The  Gate 
Reviews  are  conducted  at  the  3-star  level 
and  above,  and  attendance  is  by  invitation 
only.  Table  E2T3  of  SECNAVINST 
5000.2D  (SECNAV  2008)  contains  more 
detail  about  participants  and  topics  for 
each  Gate  Review. 

MCOTEA  is  periodically  called  upon 
to  contribute  to  or  attend  a  Gate  Review 
pertaining  to  the  T&E  metric.  Although 
this  typically  happens  at  Gate  6  (there 
are  usually  multiple  Gate  6s),  MCOTEA 
could  be  involved  in  earlier  Gate  Reviews 
as  well. 

The  key  to  success  during  a  Gate  Review  is 
to  coordinate  with  the  materiel  developer’s 
Program  Manager  (PM)  ahead  of  time  so 
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Gate  Reviews  in  the 
Acquisition  Process 


the  PM  understands  MCOTEA’s  concerns 
and  MCOTEA  understands  the  PM’s 
position  and  proposed  courses  of  action. 
Appendix  1  to  chapter  3  provides  more 
information  on  what  to  expect  at  each  gate 
review  from  the  T&E  perspective. 

Fellow  Operational  Test 
Agencies 

Each  Services  conducts  OT&E  through  its 
respective  OTA.  MCOTEA  periodically 
meets  with  fellow  OTAs  in  various  forums 
to  discuss  DOD-wide  issues  relating  to 
OT&E.  MCOTEA  also  participates  with 
one  or  more  of  these  OTAs  in  conducting 
MOT&E  (ATEC  2010). 

Joint  Interoperability  Test  Command 

For  information  technology  systems 
(including  National  Security  Systems) 
with  interoperability  requirements,  the 
Joint  Interoperability  Test  Command 
(JITC)  is  required  to  provide  system 
Net-Ready  certification  memoranda  to 
the  Director,  Joint  Staff  J-6  throughout 
the  system  life  cycle,  regardless  of  ACAT. 
JITC’s  philosophy  is  to  leverage  other 
planned  test  events  to  generate  required 
data  for  the  OSD-directed  Net-Ready 
Key  Performance  Parameter  (KPP) 
certification.  A  special  test  will  be  necessary 
only  if  other  events  do  not  provide  the 
appropriate  data. 


Communication/ 

Information  Sharing 

Navy  Enterprise  TEE  Board  of  Directors 

The  T&E  Board  of  Directors  (T&E 
BOD)  primarily  addresses  issues  of 
concern  to  the  Navy.  The  Director, 
MCOTEA  is  a  member  of  the  T&E 
BOD.  SECNAVINST  3900.44  says 
“The  Marine  Corps  members  of  this 
board  will  participate  on  a  limited  basis, 
pending  corporate  decisions  on  the 
applicability  of  the  Enterprise  concept 
of  operations  for  the  Marine  Corps” 
(SECNAV  2009).  Involvement  in  this 
Board  helps  MCOTEA  stay  abreast  of 


MCOTEA's  Fellow  OTAs 

Navy:  Operational  Test  and  Evaluation 
Force  (0PTEVF0R),  headquartered  in 
Norfolk,  VA. 

Air  Force:  Air  Force  Operational  Test 
and  Evaluation  Center  (AF0TEC), 
headquartered  at  Kirtland  AFB,  NM. 

Army:  Army  Test  and  Evaluation  Command 
(ATEC),  headquartered  in  Alexandria,  VA. 

Joint  Command:  Joint  Interoperability  Test 
Command  (JITC)  headquartered  at  Fort 
Huachuca,  AZ. 
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new  instructions,  issues,  and  direction 
from  SECNAV  and  opens  a  line  of 
communication  between  MCOTEA  and 
OPTEVFOR.  SECNAVINST  3900.44 
contains  a  list  of  all  Board  members 
(SECNAV  2009). 

N091 

N091  is  the  OPNAV  Director,  Test  and 
Evaluation  and  Technology  Requirements; 
N091  establishes  T&E  requirements  and 
issues  policy,  regulations,  and  procedures 
governing  NavyT&E.  Historically,  N091 
has  served  as  a  conduit  for  MCOTEA  to 
ASN  (RDA)  by  promulgating  directives 
from  ASN  (RDA)  to  MCOTEA  and 
including  MCOTEA  in  the  review  of 
key  pending  SECNAV  documentation. 
MCOTEA  normally  deals  with  N912,  the 
Test  and  Evaluation  division  under  N09E 

OSD  Test  Investment 
Coordinating  Committee 

The  OSD  Test  Investment  Coordinating 
Committee  (OTICC)  is  the  primary 
coordinating  structure  for  test  and  evaluation 
investment  matters  within  OSD.  The 
OTICC  advises  the  Director,  Test  Resources 
Management  Center  (TRMC)  in  oversight 
of  the  development  of  test  technology 
and  Joint  test  capabilities.  MCOTEA  is  a 
primary  member  of  the  OTICC. 

Test  Resources  Management  Center 

The  Test  Resource  Management  Center 
coordinates  DOD  test  and  evaluation 
resources  and  implements  the  annual 
DOD  Strategic  Plan  for  DOD  T&E 
Resources.  The  primary  program  for 
execution  oversight  is  the  Central  Test  and 
Evaluation  Investment  Program  (CTEIP) 
and  the  DOD  T&E  and  Science  and 
Technology  (S&T)  Programs.  CTEIP 
includes  the  Joint  Improvement  and 
Modernization  Program,  the  Resource 
Enhancement  Project,  Threat  Simulators, 
and  Target  Management  Investment 
projects.  TRMC,  in  conjunction  with 


the  OTICC,  coordinates  with  the  T&E 
Executive  Agents  for  each  Service  on  the 
review  and  submission  ofT&E/S&T 
projects  to  ensure  that  Service/Agency 
Improvement  and  Modernization  projects 
are  addressed.  MCOTEA  participates  as  a 
primary  member  on  all  of  these  program/ 
project  working  groups. 

Acquisition  Life 
Cycle  Overview 

ACAT  Designation 

One  of  the  earliest  steps  in  an  acquisition 
system’s  lifecycle  is  ACAT  designation. 

A  program’s  ACAT  is  based  on  cost  and/ 
or  MDA  designation  as  a  special  interest 
(fig.  2-3).  The  ACAT  level  determines  both 
the  level  of  review  required  by  law  and 
the  MDA’s  level  within  DOD.  All  ACAT 
programs  except  ACAT  IV  (M)  and 
Abbreviated  Acquition  Programs  (AAP) 
require  operational  testing.  MCOTEA 
participates  in  the  ACAT  determination 
process  when  the  MDA  requests 
MCOTEA’s  written  concurrence  with 
ACAT  IV (M)  or  AAP  designation. 

Evolutionary  Acquisition 

Evolutionary  acquisition  delivers  system 
capabilities  in  increments.  A  program 
executing  an  evolutionary  acquisition 
strategy  incorporates  time-phased 
requirements  into  the  system.  Block 
upgrades,  planned  product  improvements, 
and  other  efforts  that  provide  a  significant 
increase  in  operational  capability  and  meet 
an  ACAT  threshold  are  managed  as  a 
separate  increment  (DOD  2008). 

The  evolutionary  approach  recognizes 
the  need  for  incremental  improvements 
at  the  beginning  of  a  program.  The  idea 
is  to  balance  technological  maturity  with 
evolving  threats,  cost,  and  the  need  to 
get  a  capability  to  the  user  quickly.  This 
allows  the  fielding  of  an  initial,  well- 
defined,  and  significant  core  operational 
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Acquisition  Category 


Reason  for  ACAT  Designation 


Decision  Authority 


ACAT  1 

Major  Defense  Acquisition  Programs 

RDT&E  total  expenditure  >  $365  million,  or  Procurement  total  expenditure 
>  $2.19  billion,  or  MDA  designation  as  special  interest 

ACAT  ID:  USD(AT&L) 

ACAT  1C:  SECNAV,  or  if 
delegated,  ASN  (RDA)  as  the 

CAE  (not  further  delegable) 

ACAT  IA 

Major  Automated  Information  System 

Program  costs/year  (all  appropriations)  >  $32  million,  or  total  program  costs 
>  $126  million,  or  total  life-cycle  costs  >  $378  million,  or  MDA  designation  as  a 
special  interest 

ACAT  1AM:  ASD(NII)/DOD  CIO 

ACAT  IAC:  SECNAV,  or  if 
delegated,  ASN  (RDA)  as  the 

CAE  (not  further  delegable) 

ACAT  II 

Major  Systems 

Does  not  meet  the  criteria  for  ACAT  1 

Not  applicable  to  IT  System  Programs 

RDT&E  total  expenditure  >  $140  million,  or  procurement  total  expenditure 
>  $660  million,  or  ASN  (RDA)  designation  as  special  interest 

ASN  (RDA),  or  the  individual 
designated  by  ASN  (RDA) 

ACAT  III 

Weapons  Systems 

IT  System  Programs 

Does  not  meet  the  criteria  for  ACAT  II  or  above 

Weapon  system  programs 

RDT&E  total  expenditure  <  $140  million,  or  procurement  total  expenditure 
<  $660  million,  and  affects  mission  characteristics  of  ships  or  aircraft  or  combat 
capability. 

IT  system  programs 

Program  costs/year  >  $15  million  <  $32  million,  or  total  program  costs  >  $30 
million,  <  $126  million,  or  total  lifecycle  costs  <  $378  million 

Cognizant  PEO,  MCSC 
Commander,  DRPM,  or 
designated  flag  officer  or 

Senior  Executive  Service 
(SES)  official. 

ASN  (RDA),  or  designee,  for 
programs  not  assigned  to  a 

PEO,  MCSC,  or  DRPM. 

ACAT  IV(T) 

Weapons  Systems 

IT  System  Programs 

Does  not  meet  the  criteria  for  ACAT  III  or  above 

Weapon  system  programs 

RDT&E  total  expenditure  <  $140  million,  or  procurement  total  expenditure 
<  $660  million 

IT  system  programs 

Program  costs/year  <  $15  million,  or  total  program  costs  <  $30  million,  or  total 
lifecycle  costs  <  $378  million 

Cognizant  PEO,  MCSC 
Commander,  DRPM,  or 
designated  flag  officer,  SES 
official,  or  PM. 

ASN  (RDA),  or  designee,  for 
programs  not  assigned  to  a 

PEO,  MCSC,  or  DRPM 

ACAT  IV(M) 

Weapons  Systems 

Does  not  meet  the  criteria  for  ACAT  III  or  above 

OTA  endorses  in  writing  that  the  program  does  not  require  operational  test  and 
evaluation 

Not  applicable  to  IT  system  programs  (ACAT  IV  IT  programs  must  be  ACAT 

IV(T)) 

Weapon  system  programs 

RDT&E  total  expenditure  >  $10  million,  <  $140  million,  or  procurement 
expenditure  >  $25  million/year,  >  $50  million,  total  <  $660  million  total 

Cognizant  PEO,  MCSC 
Commander,  DRPM,  or 
designated  flag  officer,  SES 
official,  or  PM. 

ASN  (RDA),  or  designee,  for 
programs  not  assigned  to  a 

PEO,  SYSCOM,  or  DRPM 

Abbreviated 

Acquisition 

Programs 

Weapons  Systems 

IT  System  Programs 

OTA  endorses  in  writing  that  the  program  does  not  require  OT&E 

Does  not  meet  the  criteria  for  ACAT  IV  or  above 

Weapon  system  programs 

Development  total  expenditure  <  $10  million,  and  production  or  services 
expenditure  <  $25  million/year,  <  $50  million  total 

IT  system  programs 

Program  costs/year  <  $15  million,  and  total  program  costs  <  $30  million 

Cognizant  PEO,  MCSC 
Commander,  DRPM,  or 
designated  flag  officer,  SES 
official,  or  PM 

ASN  (RDA),  or  designee,  for 
programs  not  assigned  to  a 

PEO,  MCSC,  or  DRPM. 

(SECNAV  2008,  2.4.7;  table  E2T1)  Note:  All  funding  shown  in  FY00  constant  dollars 


Figure  2-3. 
ACAT 

Designations 
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capability  quickly  in  response  to  validated 
requirements.  This  strategy  results  in 
fielding  increased  capability  in  succeeding 
increments  as  technology  matures. 

Incremental  Testing 
Requirements 

Figure  2-4  shows  that,  aside  from 
developing  the  initial  capability,  each 
increment  starts  at  the  technology 
development  phase  and  has  its  own 
milestones  and  operational  testing 
requirements  (SECNAV  2008;  DAU 
2009).  The  CDD  defines  the  KPPs  and 
Key  System  Attributes  (KSA)  that  apply 
to  each  increment  of  Engineering  and 
Manufacturing  Development.  Each 
increment  will  complete  DT&E,  OT&E, 
and  LFT&E  as  required.  An  independent 
phase  of  OT&E  must  be  completed 
for  each  increment  before  release  to  the 
user  for  programs  requiring  OT&E.  As 
suggested  by  the  figure,  each  increment 
is  treated  individually  and  will  be  at  a 
different  phase  in  the  OT&E  process 
at  any  particular  time.  This  will  involve 
concurrent  test  planning  and  execution 
activity  for  the  different  increments  and 
may  result  in  a  higher  degree  of  complexity, 
requiring  each  increment  to  be  carefully 
tracked.  The  evolutionary  strategy  for  each 
increment  will  be  described  in  the  TEMP. 


In  general,  T&E  that  has  confirmed  the 
mission  capabilities  of  an  increment  need 
not  be  repeated  in  its  entirety  to  confirm 
that  the  subsequent  increment  continues 
to  provide  those  mission  capabilities. 
“However,  regression  testing  to  reconfirm 
previously  tested  operational  capabilities 
and/or  suitability  might  be  required  if 
the  subsequent  increment  introduces  a 
significantly  changed  hardware  or  software 
configuration,  or  introduces  new  functions, 
components,  or  interfaces  that  could 
reasonably  be  expected  to  alter  previously 
confirmed  capabilities”  (DAU  2009). 

Test  and  Evaluation  Paradigm 

The  MCOTEA  approach  to  testing 
and  evaluating  is  designed  to  maximize 
synergy  with  the  rest  of  the  Marine  Corps 
acquisition  process  consistent  with  federal 
law  and  DOD,  DON,  CJCS,  and  Marine 
Corps  guidance.  This  approach  reduces 
program  risk  and  overall  cost,  thereby 
maximizing  value  to  the  Marine  Corps  and 
DOD.  In  accordance  with  DODI 
5000.02  (DOD  2008),  MCOTEA  must 
accomplish  the  following  during  IOT&E: 

♦  determine  if  thresholds  in  the  approved 
capabilities  documents  and  COIs  have  been 
satisfied 

♦  determine  OE/OS/OSur  of  the  system 


Figure  2-4. 
Incremental 
Technology 
Development 
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under  realistic  operational  conditions, 
including  Joint  combat  operations 

♦  assess  the  effect  to  combat  operations 

♦  provide  additional  information  on  the  system’s 
operational  capabilities  and  limitations 

The  evaluation  associated  with 
accomplishing  these  tasks  is  rooted  in  a 
process  that  takes  place  throughout  the  life 
of  a  program.  MCOTEA  uses  the  results 
of  non-MCOTEA  developmental  testing 
when  appropriate  as  well  as  the  results  of 
MCOTEA’s  assessments  and  operational 
testing.  MCOTEA  accomplishes  these 
tasks  using  a  combination  of  integrated 
planning  and  frequent  testing  in 
conjunction  with  continuous  evaluation. 
MCOTEA  employs  the  DOD  definition 
of  integrated  testing:  “Integrated 
testing  is  the  collaborative  planning  and 
collaborative  execution  of  test  phases  and 
events  to  provide  shared  data  in  support 
of  independent  analysis,  evaluation,  and 
reporting  by  all  stakeholders,  particularly 
the  developmental  (both  contractor  and 
government)  and  operational  test  and 
evaluation  communities”  (Office  of  the 
Secretary  of  Defense  2008).  MCOTEA 
does  not  call  out  individual  tests  as 
being  “integrated”;  instead,  MCOTEA 
collaboratively  plans  all  test  phases  with 
the  Materiel  Developer  throughout  the 
life  of  a  program  while  maintaining  the 
independence  of  IOT.  Although  test  events 
are  collaboratively  planned  to  ensure  all 
needed  data  will  be  available,  and  some 
may  be  collaboratively  executed  (excluding 
IOT/FOT/MOT,  which  is  executed  only 
by  MCOTEA),  both  the  DT  and  the  OT 
evaluations  must  be  done  separately  and 
independently. 

Test  Relationship  to 
Evaluation 

Test  and  evaluation  are  often  thought  of 
as  a  single  process,  while  in  reality  they 
are  two  related  but  distinct  processes. 
Testing  involves  the  physical  exercising, 


by  trial  or  examination,  of  a  component, 
system,  concept,  or  approach  for  the  sole 
purpose  of  gathering  data  and  information 
regarding  the  item  under  test.  Evaluation 
seeks  to  ascertain  the  worth  of,  or  to  fix  the 
value  of,  a  component,  system,  concept,  or 
approach.  Testing  provides  a  source  of  data 
for  the  evaluation  process  that  uses  the  data 
to  derive  useful  information  about  what  has 
been  tested.  The  relationship  of  testing  to 
evaluation  is  many-to-one;  that  is,  several 
tests  may  be  required  to  support  a  single 
evaluation  (fig.  2-5). 


MCOTEA’s  System  Evaluation 
Plan  (SEP)  creates  a  framework  and 
methodology  for  evaluating  the  entirety  of 
program  data  obtained  from  assessments 
and  IOT.  The  SEP  is  intended  provide 
a  transparent,  repeatable,  and  defensible 
approach  to  evaluation  with  the  added 
benefit  of  minimizing  the  overall  cost  of 
program  testing.  Although  the  SEP  is 
an  internal  document,  MCOTEA  will 
consult  closely  with  DOT&E  to  ensure 
the  MCOTEA  evaluation  process  for 
programs  on  oversight  will  meet  DOT&E 
requirements.  In  addition,  MCOTEA 
welcomes  Program  Office  and  DC, 

CD&I  suggestions  pertaining  to  the 
SEP;  however,  in  the  final  analysis,  the 
evaluation  process  belongs  to  MCOTEA 
and  MCOTEA  is  under  no  obligation  to 
accept  these  suggestions. 

As  the  OTA  for  the  Marine  Corps, 
MCOTEA  is  charged  with  both  the 
operational  testing  and  evaluation  of 
systems.  The  purpose  of  operational  testing 
is  to  determine  how  the  system  performs 
under  test  using  production-representative 
components,  operated  and  maintained  by 
typical  users,  under  realistic  operational 
conditions.  An  operational  test  is  a  discrete 
event  that  provides  invaluable  information 
about  the  system  under  test  and  its 
expected  capabilities  and  limitations  during 
combat  operations.  It  is  a  major  input  to 
the  evaluation  of  the  system,  but  not  the 
only  input. 


Figure  2-5. 
Relationship 
of  Test  to 
Evaluation 
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MCOTEA's  involvement 
at  an  early  stage 
benefits  both  MCOTEA 
and  the  new  program 
for  the  following 
reasons: 

^Generates  COIs  at  an  early 
stage  so  system  designers 
know  the  high-level  issues 
their  system  is  intended  to 
address 

■=> Lends  operational  test 
perspective  that  aids  in 
developing  unambiguous 
requirements  that  can  be 
tested 

■=> Helps  MCOTEA  gain  better 
understanding  of  the  context 
in  which  the  capabilities 
and  requirements  were 
determined 

■=> Provides  insight  into 
potential  system  and 
operational  deficiencies 
early  in  the  program  when 
remedial  action  can  easily  be 
taken 

■=> Provides  insight  into 
potential  I0T  requirements  to 
ensure  that  range  capabilities 
and  technologies  exist  to 
meet  those  test  requirements. 
If  a  shortfall  is  recognized 
early  enough,  initiation  of  a 
test  technology  development 
program  may  be  in  order. 

■^Provides  independent 
insight  to  decision  makers  into 
the  program's  progress  toward 
meeting  the  desired  level  of 
Operational  Effectiveness, 
Suitability,  and  Survivability 


Other  tests  and  assessments  increase 
the  knowledge  about  the  system  under 
test  as  the  system  matures  during  the 
acquisition  cycle.  These  tests  also  provide 
input  to  the  overall  system  evaluation. 
During  developmental  testing,  system 
components  are  checked  to  ensure  that 
they  function  as  designed  and  the  system 
is  checked  to  ensure  that  it  meets  the 
requirements  derived  from  the  ICD/CDD/ 
CPD.  MCOTEA  generally  uses  the  data 
gathered  during  DT  to  determine  if  the 
thresholds  in  the  approved  capabilities 
documentation  have  been  demonstrated. 

In  addition,  aggregating  DT  data  over  time 
can  be  useful  in  determining  aspects  of  a 
system’s  OS.  Furthermore,  MCOTEA’s 
assessments  provide  insight  into  the  level 
of  system  maturity  and  overall  system 
capabilities  and  limitations. 

The  purpose  of  the  evaluation  is  to 
use  all  relevant  information  from  DT, 
MCOTEA’s  assessments,  operational 
testing,  relevant  M&S  results,  and 
the  results  of  any  Live  Fire  testing  to 
determine  OE/OS/OSur.  Evaluation 
involves  compilation  and  analysis  of  data 
gathered  over  the  life  of  the  program,  with 
emphasis  on  system  performance  during 
operational  testing. 

The  Evaluation  Continuum 

Advantages  of  MCOTEA’s 
Early  Involvement 

According  to  DODI  5000.02,  “T&E 
expertise  must  be  brought  to  bear  at 
the  beginning  of  the  system  lifecycle  to 
provide  earlier  learning  about  the  strengths 
and  weaknesses  of  the  system  under 
development”  (DOD  2008).  MCOTEA 
does  not  wait  until  a  full-blown  operational 
test  is  needed  to  get  involved  in  the 
program  acquisition  process.  Ideally, 
MCOTEA  involvement  begins  very  early 
in  the  acquisition  cycle.  MCOTEA’s  goal 
is  to  become  involved  in  a  new  program  as 
early  as  the  formation  of  the  Requirements 
Transition  Team  (RTT),  a  team  formed 


before  the  Materiel  Development  Decision 
(MDD)  to  facilitate  the  development  and 
transition  of  potential  requirements  into 
the  acquisition  process  (RTG  2003).  This 
early  involvement  includes  early  program 
reviews,  demonstrations,  developmental 
working  groups,  M&S  activities,  and 
other  technical  development  activities. 
According  to  SECNAVINST  5000.2D, 
“Early,  active,  and  continuous  participation 
by  test  agencies  during  the  development 
of  capabilities  documents  will  support 
effective  communication  and  common 
interpretation”  (DOD  2008). 

MCOTEA’s  goal  is  to  develop  draft  COIs 
(see  chapter  3-1)  prior  to  the  Analysis  of 
Alternatives  (AoA).The  AoA  identifies 
potential  options  to  an  MDD,  thereby 
guiding  the  Materiel  Solution  Analysis 
phase  of  acquisition. 

Having  draft  COIs  available  allows  the 
AoA  to  examine  alternatives  based  on 
the  same  high-level  Issues  the  system 
will  be  expected  to  address  throughout 
its  lifetime,  including  during  operational 
testing.  Although  other  Issues  may  also 
be  examined  during  the  AoA,  the  COIs 
should  form  the  basis  of  the  major  areas  of 
comparison  addressed  in  the  AoA. 

In  addition  to  MCOTEA’s  involvement  in 
monitoring  and  analyzing  developmental 
testing,  use  of  assessments  early  in  a 
system’s  development  can  help  to  identify 
technology  risks  and  illuminate  potential 
operational  issues.  Integrated  testing  and 
early  OAs  can  be  expected  to  emphasize 
the  use  of  prototypes.  Early  MCOTEA 
involvement  should  benefit  the  entire 
Marine  Corps  acquisition  process  while 
minimizing  the  cost  of  the  overall  program. 

Continuous  Evaluation 

The  evaluation  of  a  system  is  the  result  of 
the  accumulation  of  data  and  facts  about 
the  system  obtained  during  the  entire 
acquisition  cycle  (SECNAV  2008). This 
accumulation  of  data  starts  with  early 
research  and  developmental  testing  and 
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continues  through  IOT  and  Follow-on 
Test  (FOT).  Integrated  testing  and  early 
assessments  can  contribute  important 
contextual  information,  can  result  in 
enhanced  understanding  of  system 
capabilities,  and  can  make  significant 
contributions  to  satisfying  the  requirement 
to  examine  the  extent  to  which 
CDD/CPD  thresholds  have  been  satisfied. 
Of  course,  the  events  that  will  yield  the 
most  important  information  from  the 
system  evaluation  perspective  are  the  IOT 
and,  if  applicable,  FOT 

Figure  2-6  illustrates  how  input  from 
various  assessments  and  testing  events 
contribute  to  the  aggregated  evaluation  of  a 
system.  As  shown  in  the  figure,  in 


addition  to  operational  testing  results  near 
the  end  of  the  acquisition  cycle,  the  results 
of  observations  and  assessments  at  earlier 
stages  in  the  program  are  fed  back  to  the 
program  to  help  the  PM  identify  program 
risks.  Waiting  until  IOT  to  evaluate  a 
system  for  the  first  time  does  little  to  affect 
the  actual  design  of  the  system.  Therefore, 
MCOTEA  provides  feedback  to  the 
PM  and  MDA  periodically  during  the 
acquisition  cycle.  This  feedback  indicates  if 
a  program  is  progressing  towards  IOT  and 
identifies  potential  concerns. 


Continuous  evaluation 
increases  the  efficiency 
of  the  Marine  Corps 
acquisition  cycle  in  the 
following  ways: 

■=> Gathers  important  data  on 
most  of  the  thresholds  in  the 
capabilities  documents  before 
operational  testing 

■=>Allows  evaluation 
feedback  throughout  the 
program  focused  on  the  PM 
and  decision  maker's  needs 
and  based  on  standards 
appropriate  for  the  program's 
developmental  stage 

■=> Identifies  important  issues 
and  potential  deficiencies 
early  enough  in  the  program 
to  allow  relatively  inexpensive 
corrective  action 

■=>Enablesan  independent 
mechanism  for  tracking 
program  progress  over  time 

■=> Allows  operational 
testing  to  focus  on  COIs 
and  operational  mission 
performance  rather  than 
specification  and  threshold 
compliance 


Figure  2-6. 
MCOTEA 's  test 
and  evaluation 
process  is  a  cycle 
of  continuous 
feedback 
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Early  Identification 
of  Deficiencies 

Identification  of  system  deficiencies  is 
most  valuable  early  in  the  program.  The 
value  of  identifying  deficiencies  diminishes 
as  the  program  matures.  Alterations  to  a 
more  mature  program  are  more  difficult 
and  expensive  to  make,  whereas  assessing 
program  progress  at  early  and  intermediate 
stages  enables  the  Marine  Corps  to  adjust 
the  program  more  effectively. 

System  assessment  feedback  that  occurs 
early  in  a  program  is  different  in  nature 
from  the  evaluation  of  a  mature  program. 
MCOTEA  assesses  a  system’s  progress 
based  on  standards  appropriate  for  its 
developmental  stage.  Early  evaluation 
feedback  tends  to  be  limited  in  scope, 
but  this  feedback  builds  a  history  for  the 
program  that  shows  when  issues  were 
identified  and  how  they  were  mitigated. 
This  opens  an  additional  window  on  how 
the  program  is  maturing  as  a  function  of  time. 

Finally,  obtaining  Warfighter  feedback 
after  system  fielding  is  important  for 
optimizing  the  MCOTEA  test  process 
as  well  as  the  Marine  Corps  acquisition 
process  (see  chapter  5). 


Collaboration  Along  the 
Acquisition  Time  Line 

Throughout  the  acquisition  cycle, 
MCOTEA  brings  the  operational  testing 
perspective  to  all  milestone  assessment 
teams.  In  general  the  Materiel  Developer, 
MCOTEA,  and  DC,  CD&I  will 
participate  in  one  another’s  Subject  Matter 
Expert  (SME)  panels  throughout  the  life 
of  a  program.  The  cognizant  MCOTEA 
Test  Division  can  expect  to  participate  in 
various  Gate  Reviews  (see  page  3-12  for 
information  on  Gate  Reviews)  to  support 
the  briefing  requirements  of  PoPS  program 
criteria  pertaining  to  test  and  evaluation. 

Pre-Milestone  A 

Figure  2-7  illustrates  key  points  of 
MCOTEA’s  interaction  with  other 
agencies  before  MS  A.  Early  in  a  program’s 
life,  the  RTT  stands  up  to  facilitate  the 
transition  from  desired  capabilities  to 
an  actual  system.  Participation  in  this 
team  may  be  MCOTEA’s  first  official 
activity  on  a  new  program.  MCOTEA 
reviews  the  draft  Initial  Capabilities 
Document  (ICD)  once  it  is  written  to 
ensure  that  the  proposed  capabilities  are 
testable.  MCOTEA  also  participates 
in  working  groups  that  generate  the 
applicable  CONOPS  and  COE.  This  early 
participation  with  DC,  CD&I  enhances 


Figure  2-7. 

MCOTEA 
Observation 
and  Testing  in 
the  Acguisition 
Cycle 


MCOTEA;  DC,  CD&I; 
Materiel  Developer 
revisit  COIs 


DC,  CD&I,  Materiel  Developer, 
MCOTEA  participate  in  RTT 


MDD  AoA 


MCOTEA  &  Materiel 
Developer  observe 
CONOPS 


Preliminary  COIs  MS  A 

M  a\/  ha  hriafaH  at  CTlata  9 


Materiel  Developer,  MCOTEA 
review  and  comment 
on  ICD 


MCOTEA  &  Materiel  Developer 
review  and  comment  on  CDD 


MCOTEA;  DC,  CD&I;  Materiel  Developer 
participate  in 

milestone  assessment  team 


MCOTEA  establishes 
draft  COIs  for  AoA 

(collaborates  with  Materiel  Developer  &  DC,  CD&I) 
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MCOTEA’s  understanding  of  the  context 
in  which  the  capabilities  were  generated. 

Before  the  AoA,  MCOTEA  establishes 
draft  COIs  using  the  process  introduced 
later  in  this  chapter  and  described  in  detail 
in  chapter  3-1,  which  help  the  AoA  team 
determine  the  categories  for  comparing 
alternatives.  The  goal  is  to  use  essentially 
the  same  COIs  for  system  evaluation  from 
the  AoA  through  IOT.  After  the  AoA, 
MCOTEA  revisits  the  COIs  and  updates 
them  based  on  information  discovered 
during  the  AoA,  an  updated  understanding 
of  the  system  concept  of  operations,  and 
any  new  information  available  in  the 
capabilities  documentation. 

These  preliminary  COIs  may  be  briefed 
at  the  Gate  2  review.  The  activity  before 
MS  A  constitutes  steps  essential  to  the 
development  of  the  program  TEMP  and 
the  MCOTEA  SEP. 

Milestone  A  to  Milestone  B 


The  goal  is  to  examine  all  Attributes 
with  thresholds  in  a  way  that  meets  OT 
requirements  before  IOT,  as  well  as  to 
build  a  database  to  support  the  suitability 
determination. 


MCOTEA  CDRL  inputs  to  RFP 

Materiel  Developer  &  MCOTEA  reconcile  RFP  with  TEMP 
MCOTEA  Source  Selection  Consulting 

-  help  establish  prototype  criteria 

-  observe  prototype  demonstrations 

-  report  on  requirement  satisfaction 
from  the  operational  perspective 

-  issue  no  opinion  on  relative  performance 


The  collaborative  approach  continues 
between  MS  A  and  MS  B  (fig.  2-8).  At 
MS  A  and  B,  MCOTEA  receives  a  copy 
of  the  Acquisition  Decision  Memorandum 
(ADM).  After  MS  A,  MCOTEA  continues 
developing  a  framework  for  the  evaluation 
by  establishing  test  conditions,  determining 
any  implied  Attributes,  and  tracing  all 
Attributes  to  Sub  tasks,  Tasks,  and  ultimately 
the  COIs.  The  final  COIs  to  be  used  in 
operational  testing  may  be  briefed  at  Gate  4. 

The  Materiel  Developer  and  MCOTEA 
work  together  to  efficiently  assign  Subtasks 
and  Tasks  (and  their  associated  Issues)  for 
examination  under  specified  conditions  in 
developmental  and  operational  tests  and 
assessments  in  accordance  with  the  TEMP. 
Attributes  with  thresholds  are  also  assigned 
to  test  events  in  the  TEMP.  The  initial 
allocation  of  Subtasks  and  Tasks  to  specific 
tests  may  need  to  be  modified  based  on 
test  results  themselves;  however,  the  goal 
of  allowing  the  IOT  to  focus  on  mission 
performance  under  realistic  operational 
conditions  remains  unchanged. 


Before  issuing  the  Request  for  Proposal 
(RFP),  the  Materiel  Developer  and 
MCOTEA  ensure  that  the  RFP 
is  consistent  with  the  TEMP,  and 
MCOTEA  provides  inputs  to  the 
Contract  Deliverables  Requirements  List. 
In  particular,  MCOTEA  input  ensures 
that  any  contractor  developmental  test  data 
and  reports  are  available  for  inspection  and 
possible  inclusion  in  the  overall  evaluation 
plan.  The  Materiel  Developer  will  consult 
with  MCOTEA  when  determining  the 
source  selection  criteria.  MCOTEA  may 
participate  in  prototype  demonstrations 
associated  with  source  selection  and 
will  have  access  to  data  obtained  during 
prototype  testing.  After  prototype  testing 
MCOTEA  will  provide  input  to  the 
Materiel  Developer  from  the  operational 
test  perspective;  however,  MCOTEA  will 
not  offer  an  opinion  on  relative  candidate 
system  performance. 


Figure  2-8. 

MCOTEA  interaction 
with  system 
acquisition  between 
MS  A  and  MSB 
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Figure  2-9. 

MCOTEA 
interaction 
with  system 
acquisition  Post 
MSB 


MCOTEA  reviews  MCOTEA  identifies  major 

CPD  operational  deficiencies 


-  OAGs 

-  existing  sustainment 
databases 

-  Warfighter 


Post  Milestone  B 

After  MS  B,  in  addition  to  performing  any 
assessments,  MCOTEA  reviews  the  CPD 
(fig.  2-9)  and  continues  to  plan  for  IOT. 

After  MS  C,  MCOTEA  provides  input 
to  the  Materiel  Developer  concerning 
the  acceptance  test  criteria  used  for  each 
early  system  purchased.  MCOTEA  also 
expeditiously  alerts  the  PM  and  MDA  of 
any  major  system  or  operational  deficiencies 
discovered  during  integrated  or  operational 
testing.  Finally,  MCOTEA  seeks  feedback 
from  multiple  sources  with  an  eye  toward 
improving  MCOTEA’s  processes.  These 
sources  include  the  PM,  MDA,  Operations 
Advisory  Groups  (OAG),  databases 
designed  to  monitor  suitability  data  of 
systems  after  fielding,  and  Warfighter 
feedback  from  deployed  units. 

Obtaining  and  Using 
Developmental  Test  Data 

MCOTEA  leverages  early  testing 
opportunities  during  DT  to  maximize 
available  information  for  decision  makers 
and  to  minimize  the  risk  and  expense  of 
the  entire  testing  program. 

The  Integrated  Test  and  Evaluation 
approach  is  formulated  before  any 
developmental  testing  takes  place.  The 


T&E  approach  is  described  in  the  Program 
Manager’s  Test  and  Evaluation  Strategy, 
while  the  plan  is  described  in  detail  in  the 
TEMP  MCOTEA  participates  in  TEMP 
development  to  reflect  the  integrated  test 
approach  and  constructs  its  own  SEP  (see 
chapter  3-1)  that  details  how  data  will  be 
aggregated  and  used  in  the  final  system 
evaluation. 

MCOTEA  is  aware  of  planned  DT 
events  by  participating  in  the  T&E  WIPT 
and  can  expect  to  participate  in  the 
collaborative  planning  of  these  events.  For 
MCOTEA  to  participate  in  a  DT  event  at 
any  level,  the  draft  developmental  test  plan 
must  be  available  for  MCOTEA’s  review 
in  ample  time  for  MCOTEA  to  comment 
and  offer  suggestions  based  on  shared 
data  needs.  The  DT  team  may  or  may  not 
accept  these  suggestions,  based  on  time 
and  cost  constraints.  However,  the  PM 
should  be  aware  that  MCOTEA  testing 
requirements  will  need  to  be  satisfied  at 
some  point,  and  although  incorporating 
them  into  a  DT  event  may  raise  the  cost  of 
that  particular  test,  it  may  well  decrease  the 
overall  program  testing  cost  and  reduce  risk 
by  satisfying  MCOTEA’s  requirements 
early. 
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MCOTEA  Test  Team  Billets 
and  Best  Practices 

Following  evaluation  planning,  the  actual 
test  planning  process  begins  in  earnest. 

The  Test  Division  forms  a  test  team  for 
each  MCOTEA  program,  composed  of  an 
Operational  Test  Project  Office  (OTPO),  a 
Test  Manager  (TM),  an  Operations  Analyst 
(OA),  and  a  Data  Manager  (DM)  (fig.  2-10). 

The  Operational  Test 
Project  Officer 

The  OTPO,  usually  an  operational 
mission  area  expert  for  the  system  under 
test,  is  the  team  leader  and  is  responsible 
for  managing  the  test.  Test  project 
management  requires  staff  action  in  three 
areas:  OT&E  documentation;  system 
user-developer  coordination;  and  OT&E 
resource  management  (cost,  schedule, 
performance). 

Test  Manager 

The  TM  assists  the  OTPO  in  planning, 
executing,  and  reporting  operational  test 
events.  The  TM  often  acts  as  a 
surrogate  for  the  OTPO,  providing 
representation  at  various  meetings 
and  program  IPTs.  In  addition 
to  writing  the  draft  test  plan,  the 
TM  helps  coordinate  the  test  team, 
makes  logistical  arrangements  for 
the  test  site,  and  remains  at  the  test 
site  throughout  test  execution. 

With  each  additional  test 
managed,  specific  characteristics 
and  lessons  learned  from  previous 
tests  can  be  applied  to  the  new 
test.  However,  each  test  is  unique 
and  management  rules  will 
change  to  facilitate  the  particular 
requirements  of  the  type  of  testing 
being  managed.  Flexibility  and 
open-mindedness  are  critical  to 
managing  a  test  program  well. 

The  TM  performs  the  following 
functions: 


Reviews  the  Statement  of  Work  and 
Funding  Profile  to  ensure  adequate  funding 
and  personnel  to  accomplish  the  task 

Develops  the  Plan  of  Action  and 
Milestones  (POA&M)  in  accordance  with 
the  Statement  of  Work 

Coordinates  and  chairs  a  team  kick-off 
meeting.  During  this  meeting,  the  TM 
should  review  the  system  under  test,  discuss 
test  requirements,  and  review  the  POA&M 

Assigns  duties  and  responsibilities  to  all  test 
team  members  to  ensure  all  test  documents  are 
produced  in  accordance  with  the  POA&M 

Engages,  coordinates,  and  integrates 
with  the  Program  Management  Team, 
CD&I,  and  any  other  stakeholders  as 
early  as  possible  to  coordinate  the  test 
and  evaluation  requirements,  issues,  and 
concerns,  including  program  schedule,  risk, 
and  funding  requirements 

Holds  a  weekly  team  meeting  to  discuss 
document  development  and  monitors 
progress  through  the  POA&M. 


Figure  2-10. 
Test  Team 
Organization 
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Before  Testing 

♦  Conducts  the  initial  site  survey  and  pre¬ 
coordination  of  selected  test  sites;  this 
allows  the  early  identification  of  facilities 
required  to  support  Test  Plan  execution 

♦  Conducts  final  site  survey  and  coordination 
of  selected  test  sites  no  later  than  6  months 
before  IOT&E.This  will  confirm  the  ability 
to  execute  the  draft  Test  Plan 

♦  Conducts  a  final  test  team  meeting  no  later 
than  5  working  days  prior  to  departure  to 
ensure  all  necessary  logistics  requirements 
(test  equipment  and  test  team  members) 
arrive  on  site  as  scheduled  and  are  prepared 
to  execute  the  Test  Plan. 

During  Testing 

♦  Verifies  all  personnel  and  equipment  have 
arrived 

♦  Conducts  route  recon  from  billeting  to  the 
test  site  and  test  site  orientation 

♦  Sets  up  the  test  operations  (data  collection 
center,  support  shelters,  and  logistics 
required  to  support  the  test  team)  no  later 
than  48  hours  before  New  Equipment 
Training  (NET) 

♦  Coordinates  with  the  OTPO  additional  test 
support  as  required 

♦  Monitors  the  daily  activities  of  the  Pilot  and 
Record  Test  team  and  test  conduct  to  ensure 
the  Test  Plan  is  being  executed  as  required 

♦  Effects  changes  to  the  test  schedule  as 
requested  by  the  OTPO  and  ensures  the 
test  team  is  informed  of  those  changes 

♦  Manages  the  test  team  activities  to  ensure 
the  team  members  receive  adequate  time 
for  rest  and  recuperation  during  Pilot  and 
Record  Test  events 

After  Testing 

♦  Confirms  with  the  OTPO  that  all  data 
collection  and  Test  Plan  requirements  have 
been  met  before  closing  down  the  test  site 

♦  Confirms  that  all  test  equipment  and 
personnel  are  accounted  for  before  leaving 
the  test  site 


♦  Coordinates  the  return  of  all  personnel  and 
equipment  to  MCOTEA 

♦  Conducts  a  posttest  team  meeting  to  discuss 
the  after  actions  requirements 

♦  Ensures  that  the  Lessons  Learned  are 
discussed  and  captured  by  individual  team 
members  no  later  than  48  hours  after  return  to 
MCOTEA 

♦  Ensures  that  the  archiving  process  is 
followed  with  guidance  from  the  OTPO  and 
IT  manager. 

Operations  Analyst 

The  OA  plans  for  and  conducts  analysis 
and  evaluation  of  test  data.  This  is  done 
by  developing  the  SEP  or  the  System 
Assessment  Plan  (SAP).  The  OA  also 
assists  with  Test  Concept  development, 
test  execution,  and  data  collection  The  OA 
performs  the  following  functions: 

Document  Development 

♦  Reviews  MCOTEA  T&E  Reference 
Center  and  Lessons  Learned  to  discover 
any  similar  evaluations 

♦  Reviews  all  program  documentation  to 
understand  the  system  under  test  and  the 
missions  intended  for  its  use 

♦  Works  with  subject  area  experts  to 
understand  the  documented  and  implied 
tasks,  skills,  mission  gaps,  and  capabilities 
required  to  execute  the  intended  missions 

♦  Conducts  an  Operational  Task  Analysis 
with  SMEs  to  identify  mission  Tasks  and 
Subtasks  required  to  accomplish  the  mission 

♦  Works  with  the  TM  and  OTPO  to 
define  the  system  in  terms  of  the  required 
Tasks  and  mission  gaps  along  with  the 
boundaries  of  where  the  system  interfaces 
with  other  systems 

♦  Develops  the  evaluation  questions  (Issues) 
that  must  be  answered  to  provide  the 
determination  of  OE/OS/OSur  or  the 
specific  assessment  questions  for  a  SAP 

♦  Maps  Attributes  found  in  capabilities 
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documents  to  Tasks  and  Subtasks  to  assist 
in  ensuring  that  thresholds  are  resolved  and 
integrated  testing  opportunities  are  identified 

♦  Develops  the  Analytic  Model  by  identifying 
Measures  that  provide  answers  to  the 
evaluation  questions  (Issues)  and  aggregates 
the  Measures  into  a  model  that  represents 
mission  tasks  and  success 

♦  Develops  the  Decision  Model  that 
normalizes  the  results  from  the  Analytic 
Model  to  a  common  scale  (Mission 
Capability  Level  (MCL)) 

♦  Evaluates  all  test  data 

Before  Testing 

♦  Identifies  the  variables  associated  with  the 
Measures  contained  in  the  SEP/SAP 

♦  Identifies  cause-effect  relationships  and  the 
inputs/outputs  associated  with  the  mission 
process  flow 

♦  Ensures  the  process  flow  and  variables 
identified  are  addressed  for  each  test  event 
and  develops  an  appropriate  Design  of 
Experiments  (DOE)  and  sample  size  for 
that  event 

♦  Assists  the  test  team  with  the  FD/SC 
Charter  by  identifying  the  Mission  Essential 
Functions  and  Reliability/Survivability 
Measures  and  assisting  with  the  time 
classification  dendritic 

♦  Provides  the  Data  Manager  with  a 
concise  list  of  data  elements  required 
for  each  Measure  to  assist  with  database 
development 

♦  Works  with  the  TM  to  develop  trials  using 
the  DOE  matrix  and  available  resources 
such  has  ranges,  instrumentation  and  personnel 

During  Testing 

♦  Monitors  the  Pilot  test  to  ensure  that  trials 
are  conducted  correctly  and  data  is  collected 
and  traceable  to  each  trial 

♦  Samples  data  collected  daily  for  quality 
and  completeness,  identifying  missing  or 
incomplete  data  to  the  test  team  immediately 


♦  Assists  with  Survey  administration  and 
response  analysis 

♦  Begins  reduction  of  raw  test  data  into  the 
data  elements  required  in  the  test  design 

♦  Assists  the  test  team  with  Test  Incident  scoring 

After  Testing 

♦  Identifies  reduction  required  of  the  raw  data 
for  the  reduction  plan  section  in  the  test  plan 

♦  Identifies  potential  statistical  tests  for 
inclusion  in  the  data  analysis  method 
section  in  the  test  plan 

♦  Reduces  and  analyzes  data  in  accordance 
with  the  test  plan 

♦  Conducts  exploratory  analysis  using 
graphical  depictions  of  the  reduced  data 

♦  Verifies  the  distributions  of  the  test  results 
and  conducts  appropriate  statistical  analysis, 
verifying  assumptions  or  rules  used  by 
statistical  software  packages 

♦  Determines  confidence  bounds/intervals  to 
account  for  uncertainty 

♦  Reconstructs  trials  using  all  variables  and 
supporting  data 

Data  Manager 

DMs  support  the  OTPO,TM,  and  OA. 

The  DM  should  establish  a  good  working 
relationship  with  the  test  team  and 
the  support  personnel  to  ensure  open 
communication,  resulting  in  a  positive 
working  environment  and  a  more  efficient  test. 

The  Data  Manager  performs  a  variety 
of  duties  throughout  the  course  of  a 
program’s  lifecycle: 

Document  Development 

♦  Assists  the  TM  and  OA  in  writing  the 
SEP/SAR  The  DM  primarily  works  with 
the  OA  in  creating  the  OTA  and  the 
Mapping  Matrix 

♦  Assists  the  TM  and  OTPO  in  writing 
the  Test  Plan.  The  DM  is  responsible  for 
providing  the  data  requirements  and  the 
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data  collection  methods  and  creates  all 
data  collection  forms  and  surveys 

♦  Assists  the  TM  and  OTPO  in  preparing 
the  FD/SC  Charter  template  that  all 
stakeholders  will  use  to  categorize  any 
failures/malfunctions  that  occur  during 
test. 

♦  Assists  the  TM  and  OA  in  writing 
evaluation  reports 

♦  Attends  program-related  meetings 
(Consolidated  Review  Boards  (CRB), 
IPTs,  etc.)  and  is  responsible  for  writing 
the  meeting  minutes  if  tasked  by  the  TM 

♦  Writes  the  Data  Collection  Handbook 
before  the  test.  The  handbook  is  used 
during  data  collection  training  before  the 
Pilot  Test.  The  handbook  assists  in  training 
the  data  collectors  on  the  data  collection 
process  and  the  devices/methods  that  will 
be  used 

Before  Test 

♦  Performs  data  collection  Verification  and 
Validation 

♦  Programs  and  understands  all  data 
collection  devices  used  on  test.  Data  is 
collected  primarily  on  portable,  handheld 
electronic  devices.  Other  data  collection 
devices  may  include  stopwatches,  GPS, 
weather-reading  devices,  etc 

♦  Assists  in  organizing  and  shipping  test 
gear  to  the  test  locations.  The  DM  works 
with  the  TM  and  OTPO  to  provide 
the  S-4  with  a  list  of  required  gear.  The 
DM  ensures  that  all  gear  is  available  and 
packed  for  shipment 

♦  Conducts  the  data  collection  training  at 
the  test  site.  Training  varies  in  length  and 
format  depending  on  the  complexity  of  the 
test 

♦  Establishes  a  filing/organizational  system 
for  all  paper  forms/surveys. 

♦  Establishes  with  the  OA  a  routine  for 
downloading,  naming,  and  filing  all 
electronic  data  while  on  test,  to  ensure 
version  control  in  the  data  repository 


During  Test 

♦  Reviews  the  data  and  gives  it  to  the  OA, 
who  begins  the  data  reduction  process. 

♦  Oversees  the  entire  data  collection  process 
while  on  test.  The  DM  ensures  that  Data 
Collectors  are  accurately  collecting  the 
necessary  data  and  troubleshoots  all  data 
collection  devices  when  necessary. 

♦  Ensures  all  caveats  associated  with  data 
elements  are  properly  recorded. 

♦  Ensures  data  security  by  controlling  access 
to  recorded  data  as  well  as  read/write/edit 
privileges  associated  with  the  data. 

♦  Works  with  the  OA  on  the  test  site  to 
consolidate  and  QC  (quality  check)  all 
data  during  operational  test.  This  can  be 
conducted  after  every  trial  or  at  the  end  of 
every  test  day,  depending  on  the  format  of 
the  test. 

Supplementary  Team  Members 

The  core  test  team  (OTPO,  TM,  OA,  DM) 
are  assisted  by  the  MOIC,  Data  Collectors, 
an  IA  analyst,  a  Human  Factors  analyst, 
and  an  Accreditation  Agent  as  required  (see 
chapter  6,  section  3  (M&S)). 
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MCOTEA’s  6-Step  Test  & 
Evaluation  Process 

Step  1 .  System  Evaluation  Plan 

The  SEP  is  MCOTEA’s  overarching 
plan  for  evaluating  data  that  pertains  to  a 
system  throughout  the  life  of  the  program 
(DT  as  well  as  LFT&E  and  IOT).  The 
SEP  is  the  starting  point  of  all  IOT&E 
at  MCOTEA  and  presents  the  methods 
and  models  by  which  MCOTEA  will 
determine  OE/OS/OSur. 

The  SEP  is  a  three-part  document 
collaboratively  produced  by  the  OTPO/ 
TM  and  the  OA  assigned  to  the  program. 

Section  I  is  an  in-depth  System  Definition 
written  by  the  OTPO/TM  that  provides 
background  and  helps  the  analyst  determine 
how  best  to  evaluate  the  system  based  on  its 
mission,  crew,  components,  boundaries,  etc. 

Section  II  is  the  Evaluation  Framework, 
in  which  COIs  and  their  Measures  are 
developed.  The  development  process 
also  includes  determining  the  Tasks 
and  Subtasks  the  system  is  expected  to 
accomplish  and  additional  Issues  that 
need  answering  at  a  lower  level  than  the 
COIs.  Finally,  all  Attributes  from  the 
capabilities  documentation  are  traced  to 
one  or  more  Tasks  or  Subtasks,  creating 
the  comprehensive  framework  from  which 
system  evaluation  proceeds. 

Section  III  is  Evaluation  Methods,  in 
which  the  OA  designs  mathematically 
based  Analytic  and  Decision  Models  for 
determining  the  OE/OS/OSur  of  the 
system.  Within  section  III  is  a  depiction  of 
the  complete  evaluation  process  developed 
for  the  system  under  test. 

Step  2.  Test  Concept,  TEMP 
Input,  and  FD/SC  Charter 

With  the  SEP  in  place,  the  test  team 
begins  to  develop  details  about  the  Test 
Concept,  such  as  trial  process  flow,  sample 
size,  test  limitations,  test  resources,  required 
M&S  support,  etc.,  which  also  becomes 


input  to  part  III  of  the  TEMP.  Included  in 
this  step  are  Letters  of  Clarification  to  DC, 
CD&I,  if  necessary.  Careful  and  thorough 
development  of  the  Test  Concept  leads  to 
accurate  and  substantial  TEMP  input. 

Step  3.  Test  Planning 

MCOTEA  uses  a  mission-oriented  context 
in  operational  testing  to  relate  evaluation 
results  to  the  Warfighter’s  ability  to  execute 
missions.  Focusing  on  mission  context 
during  OT  planning  provides  a  robust 
OT  environment  and  helps  accomplish 
evaluation  goals. 

Test  planning  includes  the  following  broad 
actions,  all  of  which  are  explained  in  detail 
in  chapter  3: 

♦  Check  Lessons  Learned  Database.  The  test 
team  consults  the  Marine  Corps  Lessons 
Learned  database  (www.mccll.usmc.mil)  for 
problems  encountered  and  lessons  learned 
during  previous  operational  tests. 

♦  Establish  the  Data  Collection  plan.  The 
plan  includes  Data  Requirements  as  well  as 
Methods  for  Data  Collection,  Reduction, 
and  Analysis.  Data  may  be  quantitative  or 
qualitative  in  nature. 

♦  Design  Test  Trials.  The  test  team  designs 
trials  for  collecting  test  data,  formed  around 
the  missions  the  Marines  will  execute  using 
the  system  under  test.  Trial  methods  may 
involve  M&S;  however,  M&S  is  not  to  be 
used  as  the  only  means  of  obtaining  test  data. 

♦  Determine  Resource  Requirements.  The  test 
team  determines  resource  requirements  such 
as  funding,  required  personnel  from  the 
Operating  Forces,  number  of  test  articles, 
test  site,  instrumentation,  etc. 

♦  Confirm  Readiness  for  Test.  The 
Operational  Test  Readiness  process  ensures 
that  the  test  team  and  system  under  test  are 
ready  to  proceed  to  test.  Complete  details 
are  contained  in  chapter  3. 
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Step  4.  Operational  Test  Execution 

With  the  approved  Test  Plan  in  hand  and 
all  preparations  final,  the  test  team  arrives 
in  the  field  to  execute  operational  testing. 
Before  the  Record  Test  commences, 
however,  two  critical  steps  are  taken: 

♦  Observe  NET.  NET  is  required  for  all 
operators  and  maintainers  participating  in 
the  OT.  MCOTEA  test  team  members 
observe  NET  because  this  is  when  Tactics, 
Techniques,  and  Procedures  (TTP) 

are  taught  for  the  system  under  test.  In 
addition,  the  OTPO  and  TM  need  to  assess 
if  the  training  has  adequately  prepared 
individuals  to  proceed  to  Pilot  Test. 

♦  Execute  the  Pilot  Test.  The  Pilot  Test  is  used 
to  validate  the  data  collection  plan  and  also 
serves  as  a  rehearsal  and  readiness  check 

for  the  Record  Test.  The  OTPO/TM  allow 
adequate  time  between  the  Pilot  and  Record 
Tests  for  careful  examination  of  Pilot  Test 
data  results.  If  issues  arise  that  are  likely  to 
affect  the  Record  Test,  MCOTEA  leadership 
may  decide  to  extend  the  Pilot  Test. 

♦  Execute  the  Record  Test.  The  Record  Test 
is  the  culmination  of  all  IOT  planning.  Its 
essential  purpose  is  to  provide  the  data, 
collected  under  operational  conditions,  that 
is  required  to  evaluate  the  system  under  test. 

♦  Convene  the  FD/SC  Scoring  Conference. 
The  scoring  process  examines  the 
circumstances  associated  with  each  Test 
Incident  Report  (TIR),  and  scoring  is 
decided  by  simple  majority  vote.  If  the 
FD/SC  Conference  is  unable  to  reach 

a  conclusion,  the  Director,  MCOTEA 
decides  the  issue. 

Step  5.  Operational  Test 
Reporting 

Data  Reduction  and  Analysis 

The  DM  ensures  that  the  pedigree  of  the 
data  taken  is  maintained  and  that  all  raw 
data  taken  during  testing  is  saved  and 
available  for  access  well  after  testing  is 
complete  (see  chapter  5  for  data  archiving 
procedures).  In  many  cases  data  reduction, 


if  required,  depends  on  the  analysis 
methodology  in  use.  The  raw  data  might 
be  useful  in  future  analyses  and  should  be 
archived.  Before  leaving  the  test  site,  the 
test  team  writes  the  Test  Data  Report, 
which  provides  the  complete  raw  data  on  a 
CD  and  reports  on  test  conduct,  including 
any  Test  Limitations  or  Deviations. 

Step  6.  System  Evaluation 
and  Reporting 

The  test  team  produces  the  final  Operational 
Test  Agency  Evaluation  Report  (OER), 
which  includes  a  determination  of  OE, 

OS,  and  OSur  as  well  as  a  report  on  the 
attainment  of  thresholds  and  an  assessment 
of  the  system’s  impact  to  combat  operations. 
The  OER  also  includes  a  summary  of  all 
Major  System  and  Operational  Deficiencies 
noted  throughout  testing  and  evaluation. 

See  chapter  3-6  for  details  about  the 
reporting  process. 

Data  Archiving  and  Lessons  Learned 

MCOTEA  archives  all  test  data  and  other 
program  records  according  to  internal 
procedures  as  well  as  U.  S.  Government 
requirements.  MCOTEA  also  records 
Lessons  Learned  using  the  Marine  Corps 
Center  for  Lessons  Learned  Web  site.  See 
chapter  5  for  details. 

Process  Feedback 

MCOTEA  continuously  strives  to  improve 
its  processes  to  ensure  that  MCOTEA  tests 
and  analyses  are  relevant,  timely,  accurate, 
unbiased,  and  operationally  useful.  To  this 
end,  MCOTEA  solicits  feedback  from 
diverse  sources  as  a  means  to  improve  existing 
processes  and  identify  the  need  for  potential 
new  processes.  Any  suggestions  for  potential 
improvements  to  MCOTEA  processes 
are  forwarded  to  the  Scientific  Advisor  for 
consideration.  See  chapter  5  for  details. 

Potential  sources  of  feedback  include 

♦  MCOTEA  test  teams  and  test  Operating 
Forces 
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♦  Databases  on  deployed  systems 

♦  PM  and  MDA 

♦  OAG 

♦  Warfighters  themselves 

Types  of  MCOTEA  Tests 

Operational  Testing 

This  section  refers  to  the  steps  required  to 
execute  individual  operational  tests:  IOT, 
FOT,  and  MCOTEA-led  MOT.  This 
section  often  refers  to  IOT,  which  should  be 
viewed  as  a  final  examination  for  the  system; 
however,  wherever  IOT  is  mentioned,  the 
concepts  and  procedures  also  apply  to  FOT 
and  MCOTEA-led  MOT 

MCOTEA  uses  a  mission-oriented 
context  in  operational  testing  to  relate 
evaluation  results  to  the  impact  on  the 
Warfighter’s  ability  to  execute  missions. 
Focusing  on  the  mission  context  during 
operational  test  planning  and  execution 
provides  a  more  rebust  operational  test 
environment  and  facilitates  system 
evaluation  goals. 

Initial  Operational  Test 
and  Evaluation 

IOT&E  consists  of  the  test  itself  and  the 
subsequent  evaluation  of  test  data.  Initial 
Operational  Test  is  a  single  but  critical 
event,  while  evaluation  is  the  result  of 
a  process,  as  explained  in  detail  in  later 
chapters.  IOT  is  normally  conducted 
during  the  Production  and  Deployment 
acquisition  phase. 

In  general,  IOT  is  the  only  operational 
test  phase  required  by  Department  of 
the  Navy  policy.  In  some  cases,  when  the 
MS  C  decision  and  the  FRP  decision 
are  planned  concurrently,  IOT  may  be 
performed  during  the  Engineering  and 
Manufacturing  Development  acquisition 
phase,  prior  to  MS  C.  Characteristics  of 
IOT  are  as  follows: 

♦  Uses  production  or  production- 
representative  articles 


♦  Uses  representative  forces  (friendly  and 
opposing) 

♦  Employs  realistic  tactics,  targets,  and 
operational  environments  whenever  possible 

♦  Determines  OE/OS/OSur 

♦  May  also  support  the  decision  to  proceed 
beyond  LRIP  toFRP 

Note:  No  contractors  developing  the 
system  under  test  may  be  involved  in  the 
operation  or  maintenance  of  the  system 
during  IOT  unless  the  contractor  will 
be  involved  in  the  same  functions  when 
the  system  is  deployed  in  combat  (e.g., 
contractor  logistics  support).  If  the  system 
will  use  contractors  when  deployed, 
contractor  performance  during  IOT  will  be 
subject  to  review,  analysis,  and  evaluation  as 
part  of  the  overall  system  evaluation. 

After  IOT,  MCOTEA  evaluates  the 
data  results  along  with  other  information 
obtained  from  previous  assessments  and 
writes  an  Operational  Test  Agency  follow- 
on  Evaluation  Repport  (OFER),  which  is 
forwarded  to  the  ACMC.  After  ACMC 
approval,  the  OER  is  released  to  the  PM 
and  the  MDA. 

Follow-on  Operational  Test 
and  Evaluation 

Follow-on  Operational  Test  &  Evaluation 
(FOT&E)  is  the  operational  test  and 
evaluation  that  may  be  necessary  after  a 
successful  MS  C  or  FRP  decision.  The 
need  for  an  FOT  may  be  determined 
early  by  the  MDA  and  if  it  is,  it  should 
be  documented  in  the  TEMP.  Further 
potential  reasons  for  an  FOT&E  include 
the  following: 

♦  To  address  a  deficiency  identified  during 
system  DT  or  OT 

♦  To  ensure  that  changes  to  the  system 
since  IOT  have  remedied  previously 
recorded  deficiencies  and  have  not 
decreased  system  capability 

♦  To  refine  the  estimates,  evaluate  changes, 
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and  reevaluate  the  system  to  ensure  that  it 
continues  to  meet  operational  needs  in  a 
new  environment  or  against  a  new  threat 

FOT&E  employs  the  following: 

♦  Production  or  production-representative  articles 

♦  Typical  system  users  (Marines) 

♦  Representative  forces  (friendly  and  opposing) 

♦  Realistic  tactics  and  targets  when  possible 

♦  Operational  conditions  as  close  to  actual  as 
possible 

Note:  the  same  restrictions  on  contractor 
participation  in  test  apply  for  FOT&E  as 
for  IOT&E,  above. 

MCOTEA  evaluates  the  results  of  the  FOT 
along  with  other  relevant  information  and 
prepares  an  OFER  as  described  in  chapter  4. 

Multi-Service  Operational  Test 
and  Evaluation 

MOT&E  is  conducted  jointly  by  two  or 
more  Services.  When  designated  the  Lead 
Service,  MCOTEA  prepares  a  single 
TEMP  and  MOT  plan  in  coordination 
with  all  interested  Services  and  defense 
agencies  in  accordance  with  the  latest 
MOT&E  Memorandum  of  Agreement 
(ATEC  2010).  Like  IOT,  MOT  is  a  single 
but  critical  event,  while  evaluation  is  the 
result  of  a  process.  MOT  is  conducted  as 
follows: 

♦  uses  production  or  production- 
representative  articles 

♦  uses  appropriate  members  from  the 
operating  forces  (friendly  and  opposing) 

♦  employs  realistic  tactics  and  targets 
whenever  possible 

♦  installs  and  uses  the  system  under  test  as 
closely  as  possible  to  operational  conditions 

Note:  the  same  restrictions  on  contractor 
participation  apply  for  MOT&E  as  for 
IOT&E. 


Marine  Corps  Lead  Service 

When  the  Marine  Corps  functions  as  the 
Lead  Service  in  an  MOT&E,  MCOTEA 
is  responsible  for  accomplishing  the 
following  (not  necessarily  in  this  order): 

♦  Conduct  test  planning,  execution,  and  system 
evaluation  in  accordance  with  this  manual 

♦  Form  the  appropriate  Multi-Service  T &E  WIPT 

♦  Form  a  Test  Management  Council 
composed  of  one  senior  representative 
from  each  supporting  Service  to  arbitrate 
disagreements  that  cannot  be  solved  at  the 
T&E  WIPT  level 

♦  Participate  in  early  acquisition  activities, 
including  developmental  testing,  and  invite 
other  Service  participation  as  they  require 

♦  Issue  a  call  to  the  other  interested  Service 
OT&E  agencies  for  COIs  and  their  Service- 
unique  resource  requirements 

♦  Coordinate  action  on  the  TEMP  to  account 
for  other  Service  issues  and  inputs 

♦  Call  a  meeting  of  participating  OTA 
Test  Managers  to  assign  responsibility  for 
accomplishing  evaluation  and  test  objectives 

♦  Formulate  the  test  and  evaluation  strategy 
and  portions  of  the  TEMP  in  coordination 
with  interested  OTAs  and  the  cognizant 
Joint  Program  Office  (JPO) 

♦  Report  deficiencies  identified  in  the  system 
under  test  in  accordance  with  this  manual 

♦  Coordinate  Failure  Definition/Scoring 
Criteria  (FD/SC)  Charter  development 

MCOTEA  evaluates  the  results  of  the 
MOT  along  with  information  from 
previous  assessments  in  accordance  with  this 
manual  and  the  MOT&E  Memorandum 
of  Agreement  (ATEC  2010).  MCOTEA 
coordinates  the  evaluation  with  the  other 
Services  and  documents  the  results  in  an 
OER.  The  results  are  forwarded,  as  required, 
to  the  DOT&E,  ACMC,  MDA,  and  PM. 

Other  Service  OTA  Lead 

When  another  Service  OTA  leads  the 
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MOT&E,  Marine  Corps  inputs  are  either 
fully  integrated  within  the  TEMP  or  a 
Marine  Corps  appendix  is  included  in  the 
TEMP.  In  either  case,  the  MCOTEA 
input  should  clearly  address  unique  Marine 
Corps  issues,  requirements,  and  concerns 
with  the  planned  test  and  evaluation 
program.  This  input  provides  the  basis 
for  any  USMC-unique  testing  that 
might  be  required.  MCOTEA  leads  any 
USMC-unique  testing  and  participates 
in  other  parts  of  the  test  and  evaluation 
as  appropriate.  MCOTEA  will  conduct 
a  Marine  Corps-only  Operational  Test 
Readiness  Board  (OTRB)  before  Marines 
participate  in  an  MOT  led  by  another 
Service.  MCOTEA  will  sign  both  the 
TEMP  and  the  final  test  report  for  any 
MOT&E  that  involves  Marine  Corps  issues. 

Other  Joint  Tests 

MCOTEA  may  be  asked  to  participate 
in  Joint  Test  and  Evaluations  (JT&E) 
and  Joint  Capabilities  Technology 
Demonstrations  (JCTD).  Both  of  these 
attempt  to  address  shortfalls  in  Joint 
warfighting  capability.  To  this  end,JT&Es 
focus  more  on  developing  TTPs,  while 
JCTDs  focus  more  on  developing  new 
technologies,  hardware,  and  software. 

JTEtE 

A  JT&E  evaluates  TTPs,  concepts, 
architectures,  and  processes  to  address 
Warfighter  needs  and  issues  that  occur 
in  the  Joint  environment.  JT&Es  are 
funded  by  the  DOT&E  Deputy  Director, 
Air  Warfare  typically  for  1-3  years  (1 
year  for  a  quick  reaction  test  (QRT), 

3  years  for  a  Joint  test).  MCOTEA’s 
involvement  in  Joint  Tests  is  generally 
limited  to  Technical  Advisory  Board 
participation.  However,  MCOTEA  may 
lead  or  otherwise  participate  in  a  QRT.  The 
level  of  MCOTEA  support  for  any  given 
JT&E  is  at  the  discretion  of  the  Director, 
MCOTEA.  For  more  information  on 
JT&Es,  see  www.jte.osd.mil. 


JCTD 

A  JCTD  is  designed  to  demonstrate  a 
desired  capability  based  on  the  use  of 
mature  advanced  technologies  in  a  realistic 
environment.  JCTDs  are  initiated  by  USD 
(AT&L)  in  response  to  a  Combatant 
Commander  request.  Since  a  JCTD  is  not 
a  formal  acquisition  program,  MCOTEA 
has  no  official  requirement  to  participate. 
However,  given  that  JCTDs  can  transition 
to  a  formal  acquisition  program,  early 
participation  by  MCOTEA  may  be  in  the 
best  interest  of  the  Marine  Corps  when 
requested  and  resourced.  The  Director, 
MCOTEA  will  decide  whether  a  JCTD 
merits  MCOTEA’s  involvement  and 
the  level  of  that  involvement.  For  more 
information  on  JCTDs,  see  www.acq.osd. 
miPjctd. 

MCOTEA  Assessments 

MCOTEA  conducts  three  types  of 
assessments:  system,  intermediate,  and 
operational.  A  System  Assessment  is 
based  on  a  SAP,  while  Intermediate  and 
Operational  Assessments  stem  from  a  SEP. 
An  assessment  provides  a  “progress  report” 
on  a  system,  not  a  “final  grade,”  which 
would  be  OE/OS/OSur. 

Common  to  all  assessments  are  the 
following  characteristics: 

♦  Contractors  may  be  used  to  operate  and 
maintain  the  system 

♦  Use  of  production-representative  articles  is 
not  required 

♦  Technology  demonstrators,  prototypes, 
mock-ups,  engineering  development 
models,  or  simulations  may  be  used 

♦  OE/OS/OSur  is  not  determined 

The  results  of  any  assessment  are  sent  to  the 
PM  and  MDA  and  maybe  distributed  further 
at  the  discretion  of  the  Director,  MCOTEA. 

Complete  guidance  about  MCOTEA 
assessments  is  contained  at  the  end  of 
chapter  3. 


Types  of  Assessment 
and  Testing 
Performed  by 
MCOTEA 

System  Assessment 

•  ACATIV(M) 

•  AAP 

•  Quick  Reaction 
Assessment 

•  Non-programs  of 
record 

Intermediate  Assessment 

•  DT  Observation  (ACAT 
IV  (T)  &  above) 

Operational  Assessment 

•  Early  Operational 

•  Operational 

Operational  Testing 

•  Initial 

•  Follow-on 

•  Multi-Service 
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Top-Level  MCOTEA 
Functions 

Following  are  the  top-level  functions 
performed  by  MCOTEA  (SECNAV  2008) 
with  further  explanation  throughout  this 
manual: 

♦  Ensure  that  the  OT  of  all  ACAT  I,  IA, 

II,  III,  and  IV(T)  programs  is  effectively 
planned,  conducted,  evaluated,  and  reported 

♦  Coordinate  the  scheduling  of  resources 
for  operational  testing  requiring  Marine 
Operating  Forces  support  through  Force 
Synchronization  Conferences  and  the  Two- 
Year  Master  Test  Plan 

♦  Provide  input  to  the  TEMP,  Parts  II— IV 

♦  Prepare  an  OER  within  90  days  (but  as 
expeditiously  as  possible)  after  completing 
IOT&E  and  provide  directly  to  the  ACMC 

♦  Assist  program  acquisition  by  conducting 
Early  Operational  Assessments,  usually 
before  MS  B  and  Operational  Assessments, 
usually  before  MS  C,  on  request 


♦  Assist  program  acquisition  by  collaboratively 
planning  and  participating  in  integrated  test 
events,  observing  developmental  test  events 
and  providing  Observation  Reports,  and 
conducting  Assessments  throughout  the 
acquisition  cycle 

♦  With  the  PM,  decide  the  number  of  system 
articles  to  be  procured  for  Initial  Operational 
Testing  for  all  Acquisition  Programs  not  on 
the  OSD  T&E  Oversight  List 

♦  Coordinate  with  Marine  Operating  Forces 
and  other  commands  in  matters  related  to 
OT&E  by  publishing  a  Feasibility  of  Support 
message 

♦  Be  the  primary  interface  with  JITC  on  Joint 
interoperability  testing  conducted  during 
operational  testing 

♦  Manage  those  OSD-directed  Multi- Service 
OT&Es  for  which  the  Marine  Corps  is  tasked 

♦  Coordinate  Marine  Corps  support  for  other 
Services’  OT&Es 

♦  Effectively  represent  the  Marine  Corps  in 
all  Multi- Service  OT&E  matters 
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Test  Concept,  TEMP  Input,  and  FD/SC  Charter  Development 

Operational  Test  Planning 

Operational  Test  Execution 
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The  6-Step  Process 


This  chapter  describes  in  detail  the  6-step 
process  MCOTEA  consistently  uses  to  per¬ 
form  test  and  evaluation  once  a  new  pro¬ 
gram  has  entered  the  Activity.  Each  step  is 
presented  from  the  perspective  of  integrated 
testing;  the  Assessments  section  is  at  the 
end  of  this  chapter.  This  introduction  pres¬ 
ents  an  overview  of  the  complete  process. 

Entry  of  New  Work  into  MCOTEA 

Requests  to  Support  Early 
Collaborative  Planning 

Any  requests  for  MCOTEA’s  assistance 
in  developing  new  programs  by  external 
organizations,  including  those  that  arrive 
before  program  funding  to  MCOTEA,  are 
processed  by  the  S-2,  who  chairs  the  New 
Effort  Integrated  Product  Team  (IPT). 

The  IPT’s  purpose  is  to  determine  the 
appropriate  level  of  MCOTEA’s  support 
and  the  Division  that  will  execute  the  work. 
IPT  members  are  the  Scientific  Advisor, 
the  COT,  and  the  potential  cognizant 
Division  Head. 

Early  requests  typically  come  from  CD&I, 
the  Materiel  Developer,  or  the  RTT  in 
support  of  early  collaborative  planning  to 
include  drafting  of  COIs  and  participation 
in  the  Capabilities  Documentation  IPT 
where  MCOTEA  reviews  capabilities 
documents  and  CONOPS/Employment. 

Once  the  New  Effort  IPT  decides  to 
recommend  MCOTEA’s  involvement  in 
a  new  program,  the  S-2  generates  a  Letter 
of  Acceptance  in  collaboration  with  the 
cognizant  Division.  The  letter  describes 
MCOTEA’s  anticipated  level  of  support 
and  includes  a  Rough  Order  of  Magnitude 
cost  estimate  pending  further  program 
definition  and  funding. 

Early  and  periodic  interaction  between 
MCOTEA  Test  Divisions  and  Program 
Group  Directors/Program  Managers  is 
expected  and  encouraged  to  help  forge 
productive  working  relationships. 


Plan -Test- Report 

MCOTEA  organizes  its  test  and  evalu¬ 
ation  process  into  six  steps  (fig.  3-1), 
grouped  in  a  Plan -Test- Report  arrange¬ 
ment.  The  evaluation  process  spans  the 
entire  arrangement. 

Proper  evaluation  can  only  result  from 
the  accumulation  of  data  and  facts  about 
a  system  over  its  acquisition  life  cycle,  not 
from  a  single  operational  test.  An  overarch¬ 
ing  approach  assures  decision  makers  that 
MCOTEA’s  final  report  is  wholly  credible 
and  defensible  because  it  is  based  on  evalu¬ 
ated  test  results  spanning  the  program’s 
history. 

The  System  Evaluation  Plan  (SEP),  devel¬ 
oped  in  step  1,  is  MCOTEA’s  three-part 
plan  for  analyzing  data  from  specific  types 
of  assessments  and  operational  tests.  The 
SEP  also  “feeds”  the  Test  Concept,  the 
TEMP,  and  the  FD/SC  Charter,  devel¬ 
oped  in  step  2. 

Details  of  test  trials  and  test  logistical 
needs  are  accounted  for  in  step  3,  Opera¬ 
tional  Test  Planning,  leading  directly  to 
Test  Execution  in  step  4.  By  this  time,  all 
assessments  performed  as  part  of  integrated 
testing  are  concluded. 

Steps  5  and  6  produce  the  Test  Data 
Report  (TDR)  and  the  Operational  Test 
Agency  Evaluation  Report  (OER).  The 
TDR  provides  an  early  look  at  test  data, 
while  the  OER  analyzes  the  data  in  depth 
and  provides  decision  makers  with  an  OE/ 
OS/OSur  determination. 

These  six  steps,  grounded  in  the  scientific 
method  and  applied  consistently  across  all 
programs,  ensure  a  substantial  and  thor¬ 
ough  test  and  evaluation  process. 
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MCOTEA’s  6-step  Operational  Test  and 
Evaluation  Process 


Report 


1 


System 

Evaluation  Plan 


-  Program  Initiation 

-  SEP  Development 


Test  Concept, 

Test  and  Evaluation 
Master  Plan  Input, 

and  Failure  Definition/ 
Scoring  Criteria 
Charter 
Development 


Test  Planning 

•  Operational  Test  Plan 
and  Logistics 


4  Operational 
Test  Execution 


-  New  Equipment 
Training 

-  Pilot  Test 

-  Record  Test 

-  Posttest  Activities 


Test  Data  Report 

-  Test  Deviations 

-  Data  (unanalyzed) 


6  System 

Evaluation  and 
Reporting 

-  Final  evaluation 


Test  Data  Report 
Development 


-  Operational  Test 
Agency  Evaluation 
Report  (OER) 

or  Operational  Assessment 
Report  (OAR) 


Continuous  Evaluation  Occurs 
during  Integrated  Testing 


Figure  3-1. 
MCOTEA's  6-Step 
Process 


Integrated  Testing  Within  the 
6-Step  Process 

MCOTEA’s  primary  mission  is  OT&E, 
but  considerable  effort  is  also  devoted 
to  integrated  testing,  discussed  in  detail 
in  chapter  2.  In  terms  of  MCOTEA’s 
6-step  process,  integrated  testing  occurs 
primarily  between  steps  2  and  3,  before 
IOT  commences  (fig.  3-2).  MCOTEA 
may  use  or  perform  various  assessments 
to  provide  information  about  a  system’s 
progress  towards  IOT  or  to  gather  data  to 
fulfill  evaluation  requirements  established 
in  the  SEP.  See  the  section  at  the  end 
of  this  chapter  for  a  detailed  view  of  the 
Assessment  process. 


Types  of  MCOTEA  Assessments 

Within  the  integrated  test  process  are 
three  possible  types  of  assessments 
that  MCOTEA  can  perform:  System 
Assessment,  Intermediate  Assessment, 
and  Operational  Assessment. 

Assessments  are  performed  according 
to  a  stated  need  for  certain  types  of 
information,  as  explained  below. 

System  Assessments  pertain  to  programs 
being  tested  or  examined  at  less  than  full 
IOT,  such  as  Quick  Reaction  Assessments 
(QRA),  AAPs,  ACAT  IV(M)  programs, 
and  non-Programs  of  Record.  System 
Assessments  are  governed  by  a  SAP,  a 
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MCOTEA’s  Intermediate  and 
Operational  Assessment  Process 


Figure  3-2. 
Intermediate  and  Operational 
AssessmentProcess 
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shorter  version  of  the  SEP.  MCOTEA  uses 
this  type  of  assessment  to  answer  specific 
questions  to  address  risk  areas. 

Intermediate  Assessments  pertain  to 
programs  at  the  ACAT  IV(T)  (Test)  level 
and  above.  They  are  performed  as  a  result 
of  DT  observation  or  when  MCOTEA 
plans  and  executes  all  or  part  of  a  DT 
event.  This  can  occur  numerous  times  in  a 
program’s  life.  Intermediate  Assessments 
are  governed  by  a  SEP. 

Intermediate  Assessments  yield  Intermediate 
Assessment  Reports  (IAR).  IARs  provide 
useful  feedback  to  the  PM  and  MDA  during 
system  development  and  may  be  used  in 
support  of  Gate  Reviews. 


Operational  Assessments  demonstrate 
selected  system  performance,  with  user 
support  as  required.  An  OA  can  range  from 
a  “paper  assessment”  to  a  M&S  assessment 
to  a  physical  operational  test.  The  nature 
of  the  OA  is  described  in  the  TEMP 
and  is  governed  by  the  SEP.  An  OA  is  a 
MCOTEA-led  event. 

An  Early  Operation  Assessment  is 

similar  to  an  OA,  but  is  conducted  during 
the  Technology  Development  phase  of 
the  acquisition  cycle,  before  MS  B,  and 
is  typically  used  as  an  input  to  determine 
whether  a  system  should  continue 
development  and  proceed  to  Engineering 
and  Manufacturing  Development. 
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Step  7:  System  Evaluation  Plan 


Evaluation  Purpose 

MCOTEA’s  evaluations  support 
stakeholders  with  information  for 
pending  decisions  or  validate  decisions 
already  made.  Before  beginning  to 
develop  an  evaluation  plan,  the  evaluator 
should  understand  the  evaluation’s 
exact  purpose.  A  common  purpose  for 
a  MCOTEA  evaluation  is  to  support 
the  acquisition  process  through  the 
determination  of  OE,  OS,  and  OSur  of 
materiel  solutions. 

MCOTEA’s  conclusions  about  OE,  OS, 
and  OSur  are  considered  summative 
evaluations.  The  purpose  of  summative 
evaluation  is  to  render  a  summary 
judgment  on  a  system’s  performance 
(Scrivner  1991).  Summative  evaluations 
determine  whether  the  expectations  for  a 
system  have  been  met.  Their  findings  are 
intended  for  decision  makers  with  major 
roles  in  system  oversight.  Such  evaluations 
may  influence  significant  decisions 
about  the  continuation  of  the  system, 
allocation  of  resources,  or  restructuring. 
Therefore,  summative  evaluations  must  be 
based  on  information  that  is  sufficiently 
credible  under  scientific  standards  to 
provide  a  confident  basis  for  action  and  to 
withstand  criticism  aimed  at  discrediting 
the  results  (Rossi,  Lipsey,  Freeman  2004). 

MCOTEA  has  the  capability  to  evaluate 
non-materiel  solutions  including 
Verification,  Validation,  and  Accreditation 
(W&A)  of  simulators  and  simulations; 
training  methods;  and  TTR  These 
nonstandard  evaluations  follow  the  same 
general  process  outlined  in  this  chapter, 
even  though  terminology  and  evaluation 
questions  may  differ  depending  on  the 
evaluation’s  purpose. 


Evaluation  Paradigm:  The 
Importance  and  Benefits  of 
Continuous  Evaluation 

The  evaluation  of  a  system  for  OE/ 
OS/OSur  requires  a  wide  range  of  data  and 
information,  more  than  can  normally  be 
derived  from  a  single  test  event  (Giadrosich 
1995).  The  Defense  Acquisition  Guidebook 
recommends  “an  integrated  DT/OT/ 
LFT&E  evaluation,  using  a  phased 
approach  that  identifies  key  decision 
points  and  that  generates  timely  and 
objective  information  for  decision  makers 
on  the  system’s  demonstrated  capabilities 
to  date.”  Furthermore,  system  evaluation 
reports  should  be  prepared  in  recognition 
of  the  need  for  multiple  assessments 
of  the  performance  of  a  system  under 
development.  The  information  from  the 
evaluations  should  be  issued  periodically 
throughout  Integrated  Test  activities.  This 
information  provides  a  feedback  loop  to 
inform  systems  development  and  minimize 
the  number  of  system  faults  that  are 
discovered  in  late-stage  operational  testing 
(National  Research  Council  1998). 

Much  is  learned  about  a  system  as  it 
progresses  through  the  developmental 
cycle.  With  a  continuous  evaluation 
approach  the  independent  evaluator 
can  assess  the  system’s  progress  against 
standards  appropriate  for  that  phase  of 
development.  Early  information  about 
achievement  of  performance  specifications 
is  useful  to  the  decision  maker  when  the 
evaluation  and  information  are  provided 
with  sufficient  time  to  react  and  affect 
changes  in  design.  The  key  point  is  that 
saving  the  evaluation  of  developmental  test 
data  for  independent  evaluation  later  in  the 
developmental  cycle  when  the  operational 
testing  occurs  negates  the  point  of  the 
early  information;  information’s  usefulness 
diminishes  as  time  passes.  In  short,  to 
enable  more  timely  use  of  information, 
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MCOTEA’s  independent  assessments 
(and  reporting)  should  occur  as  closely 
as  possible  to  the  test  events  generating 
the  results.  An  increase  in  frequency  of 
communication  between  the  independent 
evaluator  and  the  decision  maker  will 
increase  the  likelihood  of  positive  changes 
in  a  system’s  design. 


Part  I.  Define  System 

To  understand  the  evaluation  process,  the 
evaluator  must  understand  the  system  and 
its  progression  through  the  development 
process.  A  system  is  an  assemblage  or 
combination  of  elements  or  parts  forming 
a  complex  or  unitary  whole  (Blanchard, 
Fabrycky  1990). 

The  system  is  defined  as  the  Marine 
unit/crew  and  their  equipment,  which 
includes  the  materiel  solution  that  will 


be  used  to  accomplish  missions.  This  is 
the  case  even  if  the  exact  composition  of 
the  materiel  solution  is  not  known  when 
developing  the  SEP.  The  description  of 
the  materiel  solution  and  the  system 
users  will  most  likely  come  from  the 
capabilities  documents  or  urgent  needs 
statements.  These  documents  provide 
descriptions  of  the  materiel  solutions  to 
include  the  necessary  KPPs,  KSAs,  and 
other  Attributes  for  the  system  that  are 
necessary  to  design  and  build  a  materiel 
solution.  These  documents  also  provide  the 
quantities  of  systems 
to  be  fielded  and  the 
units  who  will  receive 
them. 

Systems  are  composed 
of  components, 
attributes,  and 
relationships  described 
as  follows: 

♦  Components  are 
the  operating  parts  of 
a  system  consisting 
of  input,  process,  and 
output.  Each  system 
component  may 
assume  a  variety  of 

values  to  describe  a  system  state  as  set  by 
control  action  and  one  or  more  restrictions 
(Blanchard,  Fabrycky  1990). 

♦  Attributes  are  the  properties  or  discernible 
manifestations  of  the  components  of 

a  system  (Blanchard,  Fabrycky  1990). 
Attributes  characterize  the  system;  DOD 
further  defines  them  as  a  testable  or 
measureable  characteristic  that  describes  an 
aspect  of  a  system  or  capability  (Defense 
Acquisition  University  2005). 

♦  Relationships  are  the  links  between 
components  and  attributes. 

A  system  is  a  set  of  interrelated 
components  working  toward  some 
common  objective.  The  objective  or 
purpose  of  a  system  must  be  explicitly 
defined  and  understood  so  that  system 
components  provide  the  desired  output  for 
each  given  set  of  inputs. 


System  Evaluation  Plan 

The  SEP  is  MCOTEA’s  three-part  plan 
for  analyzing  data  from  Intermediate 
and  Operational  Assessments  and  Tests. 
Part  I  defines  the 
system,  including  the 
crew  or  unit  that  is 


Benefits  of  Multiple  Evaluation  Reporting 


intended  to  receive 
the  system.  Part  II 
is  the  Evaluation 
Framework, 
which  identifies 
the  Evaluation 
Questions  (COIs 
and  Issues)  that  must 
be  answered  along 
with  their  Standards 
and  Measures.  The 
Evaluation  Framework 
also  provides 
the  traceability 

of  Attributes  back  to  the  capabilities 
documents.  Part  III  describes  the 
evaluation  methods  that  will  be  used 
to  evaluate  the  results,  including  any 
aggregation  techniques  used  in  the 
evaluation  process.  (See  chapter  4  for  a 
detailed  sample  template.) 


Reporting  of  information  is  timelier  to  the 
decision  maker 


Evaluation  products  themselves  are  more 
focused  on  a  smaller  set  of  evaluation  topics 
at  greater  depth 

Evaluation  level  can  focus  on  the  decision 
maker's  needs  at  that  phase  of  development 

System  evaluation  subsequent  to 
operational  testing  can  focus  on  mission 
performance  rather  than  a  combination 
of  specification  compliance  and  mission 
performance 
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Systems  Viewpoint 


Figure  3-1-1. 
Hierarchical 
Depiction  of  a 
System 


The  definition  of  a  system  is  not  complete 
without  considering  its  position  in  the 
hierarchy  of  systems.  Every  system  is  made 
up  of  components,  and  any  component  can 
be  broken  down  into  smaller  components. 
If  two  hierarchical  levels  are  involved  in 
a  given  system,  the  lower  is  conveniently 
called  a  subsystem  (Blanchard,  Fabrycky 
1990)  (fig.  3-1-1). 

In  any  particular  situation  it  is  important 
to  define  the  system  under  consideration 
by  specifying  its  limits  or  boundaries. 
Everything  that  remains  outside  the 
boundaries  of  the  system  is  considered  to 
be  the  environment.  However,  no  system  is 
completely  isolated  from  its  environment. 
Material,  energy,  and/or  information  must 
often  pass  through  the  boundaries  as  inputs 
to  the  system.  In  reverse,  material,  energy, 
and/or  information  that  passes  from  the 
system  to  the  environment  is  called  output. 
That  which  enters  the  system  in  one  form 
and  leaves  the  system  in  another  is  usually 
called  throughput  (Blanchard,  Fabrycky 
1990). 

The  systems  viewpoint  looks  at  the  system 
from  the  top  down  rather  than  from  the 
bottom  up.  Attention  is  first  directed  to  the 
system  as  a  black  box  that  interacts  with  its 
environment.  Next,  attention  is  focused  on 
how  the  smaller  black  boxes  (subsystems) 
combine  to  achieve  the  system  objective. 

The  lowest  level  of  concern  is  then  with 
individual  components.  Focusing  on 
systems,  subsystems,  and  components 
in  a  hierarchy  forces  consideration  of 
all  pertinent  functional  relationships. 
Components  and  attributes  are  important, 
but  only  in  that  the  purpose  of  the  whole 
system  is  achieved  through  the  functional 
relationships  linking  them  (Blanchard, 
Fabrycky  1990). 


Part  II.  Evaluation  Framework 

[Before  beginning  the  Evaluation 
Framework,  the  test  team  should  check  the 
MCOTEA  Lessons  Learned  database  for 
helpful  suggestions.] 

At  the  top  level  of  the  hierarchy  are  the 
missions  (COIs).  Subordinate  to  the  COIs 
are  Tasks,  followed  by  Subtasks,  etc. 

For  OE/OS/OSur  evaluations,  each  Task 
and  Subtask  represents  an  action  to  be 
accomplished  by  equipment,  personnel, 
facilities,  software,  or  any  combination 
thereof.  Each  Task  and  Subtask  also 
represents  a  potential  evaluation  question. 
As  seen  in  figure  3-1-2,  the  evaluation 
hierarchy  flows  from  left  to  right.  Added 
to  that  is  a  top-to-bottom  addition  of 
suitability  and  survivability  characteristics 
as  appropriate  under  each  Task  and 
Subtask.  This  hierarchy  of  COIs,  Tasks, 
Subtasks,  and  their  associated  Issues  forms 
the  basis  for  the  Evaluation  Framework. 

Missions  form  the  basis  for  the  COIs  used 
to  resolve  OE/OS/OSur.  Tasks,  Subtasks, 
and  suitability/survivability  characteristics 
form  the  basis  for  the  remainder  of  the 
evaluation  questions  (i.e.,  Issues)  that 
support  COIs.  Answering  the  Issues 
associated  with  these  Subtasks  and  Tasks 
at  early  stages  of  system  development,  if 
possible,  provides  assurances  to  the  decision 
maker  that  the  system  is  progressing  as 
expected.  Logically  speaking,  it  is  desirable 
to  demonstrate  the  capability  at  a  Subtask 
level  before  attempting  the  Task  level. 

Operational  Task  Analysis 

MCOTEA  uses  Operational  Task  Analysis 
(OTA)  as  the  analytic  backbone  of  the 
Evaluation  Framework.  Task  Analysis 
supports  evaluations  by  breaking  down 
complex  evaluation  problems  into  more 
manageable  parts.  OTA  provides  a 
disciplined  method  for  developing  the 
framework  for  evaluation  questions  below 
the  level  of  OE/OS/OSur.  OTA  is  top- 
down  and  mission-based.  The  methodology 
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that  follows  can  also  be  applied  to 
evaluations  of  AAPs,  ACAT  IV(M)s, 
QRAs,  and  other  non-programs  of  record 
performed  by  MCOTEA. 

Identify  Missions 

The  first  step  in  identifying  applicable 
missions  is  to  start  with  the  system’s 
capabilities  documentation  supplemented 
by  the  Marine  Corps  Task  List.  An  SME 
panel  can  be  helpful  in  determining 
applicable  missions  for  the  system.  The 
ultimate  goal  for  identifying  missions  is  to 
develop  them  into  the  COIs. 

Identify  Tasks 

The  next  step  in  the  top-down  analysis  is  to 
identify  the  fundamental  Tasks  the  system 
is  expected  to  accomplish  in  each  mission. 
These  Tasks  constitute  the  discrete  actions 
that  must  occur  to  accomplish  the  mission 
(including  suitability  characteristics 
such  as  maintenance,  transportation,  and 
storage).  These  Tasks  are  founded  in  the 
capabilities  the  system  is  intended  to 
address;  therefore,  the  existing  capabilities 
documentation  is  consulted  initially.  In 
fact,  the  capabilities  documentation  may 
state  some  Tasks  explicitly.  Since  this  step 
is  accomplished  early,  the  capabilities 
documentation  can  be  supplemented  with 
other  authoritative  sources  (see  sidebar). 
Determining  the  Tasks  lays  the  foundation 
for  the  Evaluation  Framework.  The  focus  at 
this  point  should  be  on  the  Tasks  that  are 
required  as  opposed  to  how  the  Tasks  will 
be  accomplished.  Determining  how  a  Task 
is  accomplished  is  the  Materiel  Developer’s 
responsibility  (when  it  comes  to  the 
materiel  solution)  using  operational  TTPs. 
At  the  end  of  this  step,  all  Tasks  by  nature 
will  be  tied  to  at  least  one  parent  COI. 

Identify  Lower-Level  Subtasks 

At  this  point,  the  Tasks  are  subdivided  into 
lower-level  Subtasks.  Like  Tasks,  these 
supporting  Subtasks  constitute  the  discrete 
actions  that  must  occur  to  accomplish  the 
Task.  Some  Subtasks  may  be  associated 


with  more  than  one  Task;  these  should  be 
listed  with  each  appropriate  Task.  Subtasks 
are  a  means  of  identifying  what  operators 
must  do  to  accomplish  their  missions,  but 
at  a  lower  level  of  indenture  than  Tasks. 

As  with  the  Tasks,  all  Subtasks  must  be 
rephrased  into  a  question  (an  Issue)  to 
clarify  the  evaluation’s  intent.  It  may  be 
necessary  to  go  another  level  deeper  into 
the  Subtask  hierarchy  (the  Sub-Subtask), 
but  in  general,  the  first  level  of  Subtask 
should  suffice.  At  the  end  of  this  step,  all 
Subtasks  will  be  tied  by  nature  to  at  least 
one  parent  Task. 

Figure  3-1-3  (next  page)  illustrates  a 
completed  OTA  in  block  diagram  format. 
Block  diagrams  efficiently  document  the 
decomposition  of  missions.  OTA  block 
diagrams  are  set  up  left  to  right  so  the  top 
of  the  hierarchy  is  at  the  far  left.  At  the  top 
of  the  hierarchy  is  the  system,  defined  as 
the  Marine  unit/crew  and  their  equipment, 
which  includes  the  materiel  solution  that 
will  be  used  to  accomplish  missions.  This 
is  the  case  even  if  the  exact  composition  of 
the  materiel  solution  is  not  yet  known.  The 
remaining  blocks  are  the  Missions  (COIs), 
Tasks,  and  Subtasks  (Issues). 

The  OTA  is  a  working  document,  and 
given  its  potential  size,  may  be  more 
efficiently  used  electronically  rather 
than  on  paper.  In  any  case,  although  the 
document  is  not  a  printed  part  of  the  SEP, 
it  must  be  available  for  use  and  inspection 
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Evaluation 

Hierarchy 


Other  Sources  for 
Task  Identification 

•  Concept  of  Operations 

•  Concept  of  Employment 

•  Universal  Naval 
Task  List  and/or  the 
Universal  Joint  Task  List 

•  Mission  Essential  Task 
Lists  of  units  that  will 
employ  the  system 
under  test  or  currently 
employ  similar  existing 
systems 

•  Mission  documentation 
containing  relevant 
tactics,  techniques,  and 
procedures 

■  Training  manuals  and 
battle  books 

•  Subject  Matter  Expert 
panels 
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CO/s 
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Figure  3-1-3. 
A  completed 
Operational  Task 
Analysis,  which  is 
always  completed  in  a 
block  diagram 
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in  the  official  SEP  files.  In  addition,  the 
OTA  must  be  placed  in  MCOTEA’s  Test 
and  Evaluation  Reference  Center. 

Developing  Evaluation  Questions 

The  next  step  in  any  evaluation  is 
to  develop  the  evaluation  questions. 
MCOTEA  uses  some  standard  evaluation 
questions  for  acquisition  programs  that 
require  OE/OS/OSur  to  be  determined 
(see  callouts  in  the  following  pages).  In 
addition,  MCOTEA  performs  a  variety 
of  nonstandard  evaluations  to  support 
an  array  of  decisions  depending  on 
stakeholder  needs.  The  discussion  that 
follows  generally  applies  to  any  type  of 
evaluation  that  MCOTEA  might  perform; 
however,  the  determination  of  OE/OS/ 
OSur  is  only  associated  with  IOT,  FOT,  or 
MOT. 

Evaluation  Questions 

OE/OS/OSur,  COIs,  and  lower-level 
Issues  are  all  generically  termed  evaluation 
questions  in  this  chapter.  These  represent 
operational  questions  that  must  be 
evaluated.  The  determination  of  OE/ 
OS/OSur  represents  system  aggregation 
questions  across  all  required  missions. 

COIs  are  mission-level  questions,  while 
Issues  correspond  to  questions  based  on 
the  Tasks  and  Subtasks  associated  with  the 
system  as  well  as  Issues  associated  with 
aggregated  suitability  (e.g.,  Reliability, 
Availability,  Maintainability,  etc.)  and 
survivability  concerns.  If  a  system  is  found 
to  be  Not  OE,  OS,  or  OSur,  the  Issues  help 
to  determine  why. 

Characteristics  of  Evaluation 
Questions 

Evaluation  questions  should  be  operational 
in  nature,  observable,  and  testable  (Defense 
Acquisition  University  2009  and  Clemen, 

Reilly  2001).  Furthermore,  evaluation  questions 
must  be  answerable;  they  must  involve 
performance  dimensions  that  are  sufficiently 
specific,  concrete,  practical,  and  measurable  so 
that  meaningful  information  can  be  obtained 


about  their  status  (Rossi,  Lipsey,  Freeman 
2004  and  Clemen,  Reilly  2001). 

Formulating  unanswerable  evaluation 
questions  without  realizing  it  is  easy  to 
do.  For  example,  “Does  the  EFSS  provide 
effective  fire  support  to  the  MAGTF?”  is 
ambiguous:  what  does  “effective”  mean? 
How  would  an  evaluator  determine 
“effective  fire  support”?  Evaluation 
questions  may  also  include  so  few 
observable  indicators  that  little  can  be 
learned  about  them.  For  a  question  to  be 
answerable  it  must  be  possible  to  identify 
some  evidence  (observables)  that  can 
realistically  be  obtained  and  that  will  be 
credible  as  the  basis  for  the  answer.  Finally, 
the  distinguishing  feature  of  an  evaluation 
question  is  its  relationship  to  performance 
and  its  association,  at  least  implicitly,  with 
some  criteria  by  which  that  performance  can 
be  judged  (Rossi,  Lipsey,  Freeman  2004). 

Top-Down  Et  Mission-Based 

OT&E  follows  a  basic  pattern  of  reasoning 
in  its  practice  of  evaluation.  The  Defense 
Acquisition  Guidebook  recommends 
that  evaluators  focus  on  the  mission  that 
a  unit  or  crew  will  accomplish  when 
equipped  with  a  system  and  identify 
operational  capabilities  critical  to  mission 
accomplishment  (Defense  Acquisition 
University  2009).  Doing  so  starts  a  “top- 
down”  methodology  leading  to  COIs, 
Issues,  MOEs,  critical  LFT&E,  and  other 
evaluation  Issues,  Measures  of  Performance 
(MOP),  and  data  requirements. 

OE/OS/OSur  and  Mission 
Capability  Level 

MCOTEA  supports  acquisition  programs 
by  performing  evaluations  to  determine 
OE/OS/OSur.  The  conclusions  derived 
for  OE/OS/OSur  are  the  direct  result  of 
a  systematic  means  for  determining  the 
Mission  Capability  Level  (MCL)  that 
corresponds  to  each  mission  the  system  will 
perform. 

MCL  is  used  for  all  systems  being 
evaluated  for  OE/OS/OSur.  Determining 


Uses  of  Operational  Task 
Analysis 

•Issues  for  Evaluation 
Framework 

•  Basis  for  mission  essential 
functions 

•  Tasks  necessary  for  training 
evaluation 

•Defining  the  process  flow 
for  trials 
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Performance 


Suitability 


Survivability 


Figure  3-1-4. 
OE  Hierarchy 


MCL  is  not  required  by  law  or  directive, 
but  it  provides  a  systematic  means  of 
arriving  at  the  required  conclusions  for 
OE/OS/OSur. 

A  determination  of  MCL  expresses  to 
the  decision  maker,  on  a  by-mission  basis, 
the  level  of  mission  capability  that  can 
be  expected  of  the  system  for  a  particular 
mission.  The  MCL  can  also  be  used  for 
comparison  with  other  systems  scored  on 
the  same  scale  using  the  same  analytic 
model. 

OE/OS/OSur  Interrelationship 

OE,  OS,  and  OSur  are  related 
hierarchically  as  seen  in  figure  3-1-4.  OE  is 
achieved  through  a  combination  of  factors 
to  include  the  performance  of  the  system 
coupled  with  its  suitability  and  survivability 
characteristics. 

Examples  of  requirements  for  mission 
effectiveness  can  include  the  following: 

♦  System  is  deployable  to  the  mission  theater 
(suitability) 

♦  Operators  know  how  to  use  the  system 
properly  (suitability) 

♦  System  performs  as  expected  (performance) 

♦  System  does  not  adversely  affect  other 
mission  equipment  (suitability) 

♦  System  does  not  create  a  vulnerability  to  its 
operators  or  the  operators  of  other  systems 
(survivability) 

Operational  Effectiveness 

OE  is  an  expression  of  the  systems  overall 
ability  to  accomplish  its  missions  by  typical 
users  in  the  environment  planned  or  expected 
for  operational  employment.  Considerations 
include  organization,  doctrine,  tactics, 
system  performance,  suitability,  survivability, 
vulnerability,  and  threat. 

OE  forms  the  first  evaluation  tier  just 
above  the  MCL  of  the  operational  missions 
associated  with  the  system.  MCOTEA  is 
required  to  determine  OE  for  systems  that 


require  IOT  by  law.  OE  is  determined  by 
measuring  the  effects  or  outcomes  of  the 
missions  where  a  system  under  evaluation 
is  being  employed.  The  effect  is  unique  for 
each  system  and  depends  on  the  missions 
in  which  the  system  is  employed. 

Following  is  the  standard  first-tier 
evaluation  question  for  Operational 
Effectiveness: 

Is  the  Operational  Effectiveness  of  the 
XXX  system  adequate  to  achieve  an 
average  Mission  Capability  Level  score  of 
at  least  80  out  of  100? 


Operational  Suitability 

OS  is  the  degree  to  which  a  system  can  be 
placed  and  sustained  satisfactorily  in  field 
use  considering  the  following  (Defense 
Acquisition  University  2005): 


♦  availability 

♦  compatibility 

♦  transportability 

♦  interoperability 

♦  reliability 

♦  wartime  usage  rates 

♦  maintainability 

♦  safety 


♦  human  factors 

♦  habitability 

♦  manpower 

♦  logistics 
supportability 

♦  environmental 
effects 

♦  documentation 

♦  training 
requirements 


Following  is  the  standard  second-tier 
evaluation  question  for  Operational 
Suitability: 

Is  the  Operational  Suitability  of  the  XXX 
system  adequate  to  achieve  an  average 
Mission  Capability  Level  score  of  at 
least  80  out  of  100  when  Performance 
and  Survivability  are  held  constant  at 
threshold  levels? 


OS,  like  performance,  forms  the  basis  for  the 
second  tier  of  the  evaluation  questions  below 
OE.  MCOTEA  is  required  to  determine  OS 
for  systems  that  require  IOT  by  law. 
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OS  is  determined  by  measuring  the 
suitability  characteristics  of  the  system 
and  then  determining  what  impact,  if  any, 
these  characteristics  have  on  the  effects  or 
outcomes  of  the  missions. 

Operational  Survivability 

OSur  is  the  capability  of  a  system  and  its 
crew  to  avoid  or  withstand  a  manmade 
hostile  environment  without  suffering 
an  abortive  impairment  of  its  ability 
to  accomplish  its  designated  mission 
considering  the  following  (Defense 
Acquisition  University  2005): 

♦  electromagnetic  environmental  effects 

♦  susceptibility 

♦  vulnerability 

♦  Information  Assurance 

♦  Chemical,  Biological,  Radiological,  and 
Nuclear  survivability 

Following  is  the  standard  second-tier 
evaluation  question  for  Operational 
Survivability: 

Is  the  Operational  Survivability  of  the 
XXX  system  adequate  to  achieve  an 
average  Mission  Capability  Level  score  of 
at  least  80  out  of  100  when  Performance 
and  Suitability  are  held  constant  at 
threshold  levels  ? 


According  to  the  OSD  “Typically, 
survivability  testing  for  information 
and  business  systems  will  be  based  on 
information  assurance,”  (OSD  2010). 
MCOTEA  interprets  this  to  mean 
the  OSur  component  of  the  OE  for 
information  and  business  systems  is  based 
on  the  security,  integrity,  availability, 
authentication,  and  non-repudiation  of  the 
data  that  the  system  comprises. 

Like  performance  and  suitability,  OSur 
forms  the  basis  for  the  second  tier  of 
the  evaluation  questions  below  OE. 
MCOTEA  is  required  to  determine 
OSur  for  systems  that  require  IOT  by 
DOD  instruction  (DOD  2008).  OSur  is 


determined  by  measuring  the  survivability 
characteristics  of  the  system,  assuming 
realistic  friendly  and  threat  tactics,  and 
then  determining  what  effect,  if  any, 
these  characteristics  have  on  the  effects  or 
outcomes  of  the  missions. 

Critical  Operational  Issues 

The  system’s  operational  activities  (i.e., 
missions)  form  the  basis  for  the  COIs.  The 
goal  is  to  obtain  an  initial  set  of  COIs  early 
enough  so  they  are  available  for  use  by  the 
AoA.  After  additional  review,  the  COIs 
will  eventually  be  used  in  mission-based 
testing  to  help  determine  OE/OS/OSur. 

COIs  should  be  stated  generally  in  most 
cases  but  can  be  written  more  specifically 
when  a  test  is  relatively  simple.  For 
example,  a  COI  for  a  complex  test  of  the 
Ground/Air  Task  Oriented  Radar  (G/ATOR) 
asks,  “Can  the  operators  using  the  G/ATOR 
system  perform  Air  Surveillance  and 
Control  of  Aircraft?” 

For  a  relatively  straightforward  test 
such  as  for  the  Logistics  Vehicle  System 
Replacement  (LVSR)-Tractor,  a  COI 
reads,  “Can  the  operators  using  the  LVSR 
Tractor  MKR16  in  line-haul  operations 
achieve  Line-Haul  Performance  of  5.7 
hauled-equipment-tons  per  hour?” 

Issues 

Evaluations  are  focused  on  answering 
questions.  Issues  are  defined  as  any  aspect 
of  the  system’s  capability,  either  operational, 
technical,  or  other  that  must  be  questioned 
before  the  system’s  overall  military  utility 
can  be  known  (OSD  2008).  Issues  in  the 
evaluations  are  categorized  in  two  basic 
ways:  Tasks/Subtasks  and  suitability/ 
survivability. 

Tasks  and  Subtasks 

Tasks  and  Subtasks  are  a  means  of 
identifying  what  the  operators  need  to  do 
to  accomplish  their  missions.  All  Tasks  and 
Subtasks  result  in  questions  (i.e.,  Issues) 
to  clarify  the  evaluation’s  intent.  See  the 
previous  sections  in  this  chapter  on  Tasks 
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and  Subtasks  to  review  the  details  of  their 
characteristics. 

Suitability  and  Survivability 

Addressing  suitability  and  survivability 
within  the  Evaluation  Framework  rather 


than  in  a  separate  dendrite  helps  illuminate 
and  determine  their  overall  impact  to 
effectiveness  at  the  mission  level.  Suitability 
and  survivability  are  comprehensively 
examined  by  progessing  through  the 
hierarchy,  beginning  at  the  Subtask  and 
moving  to  the  Task  level. 

Not  all  Subtasks  will  result  in  an  evaluation 
question.  Some  Subtasks,  especially  at 
lower  levels  of  indenture,  may  not  apply 
to  the  evaluation  of  the  materiel  solution. 
However,  leaving  them  in  the  framework 
is  useful  to  examining  suitability  and 
survivability. 

For  example,  Subtasks  required  for  mission 
accomplishment  but  that  do  not  apply 
to  the  materiel  solution  can  be  used  to 
identify  equipment  and  actions  pertaining 
to  interoperability  and  compatibility.  Using 
the  sniper  rifle  with  scope  and  the  weather 
gauge  as  a  simple  example,  information 
from  the  weather  gauge  must  be  exchanged 
with  the  scope  on  the  rifle  for  the  sniper  to 
compensate  for  weather  effects  on  ballistics. 
Therefore,  it  is  important  to  validate  the 
interoperability  of  the  two  to  ensure  task 
accomplishment. 

Another  reason  to  include  suitability  and 
survivability  in  the  Evaluation  Framework 
involves  their  relationship  with  OE. 

A  simple  example  of  this  dependent 
relationship  from  the  suitability  perspective 
is  as  follows:  if  target  hits  are  the 
desired  effect  for  a  rifleman,  but  the  rifle 
malfunctions  (Reliability),  then  the  effect 
cannot  be  achieved. 

From  the  survivability  perspective,  if  the 
rifle  has  a  highly  reflective  surface  that 
readily  reveals  the  rifleman’s  position  to 
the  enemy,  and  the  rifleman  is  shot  before 
accomplishing  the  mission,  the  desired 
effect  cannot  be  achieved. 

Figure  3-1-5  illustrates  the  incorporation 
of  applicable  suitability  and  survivability 
characteristics  for  a  single  Subtask,  “acquire 
target. ’’This  process  should  be  repeated  for 
every  Task  and  Subtask  in  the  Evaluation 
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Framework  to  identify  potential  suitability 
and  survivability  Issues  that  can  affect  Task 
or  Subtask  accomplishment. 

Determine  Level  of  MElS 
Support  Required 

An  integral  part  of  evaluation  planning  is 
to  determine  if  M&S  support  is  needed 
and  how  it  will  be  used.  At  this  point, 
the  test  team  must  have  a  general  idea 
of  the  data  that  can  be  generated  from 
testing.  If  additional  data  will  be  required 
for  situations  or  environments  that 
cannot  be  tested  because  of  limited  test 
asset  availability,  lack  of  time,  test  range 
limitations,  cost,  or  safety  considerations, 
M&S  might  be  used  to  supply  this  data. 
Early  in  the  SEP  process,  the  test  team 
must  decide  the  candidate  applications  for 
M&S  support  (see  chapter  6). 

Construct  Evaluation  Standards 
and  Measures 

Any  question  to  be  evaluated  needs  two 
things:  a  standard  for  determining  worth 
or  value  and  a  method  of  measurement. 

The  process  of  identifying  standards  begins 
with  mapping  system  Attributes  to  the 
Tasks  and  Subtasks  in  the  OTA  diagram. 
The  process  ends  when  each  COI  and 
Issue  has  a  clearly  defined,  unambiguous 
standard  for  performance  that  can  be 
observed,  understood,  and  measured. 


Developing  Standards 

The  word  “standard”  is  used  generically  to 
refer  to  thresholds  or  other  defined  ranges 
of  acceptable  performance.  Thresholds  are 
defined  as  a  minimum  acceptable  operational 
value  below  which  the  utility  of  the  system 
becomes  questionable  (CJCS  2005). 
Standards  for  Performance 
and  Conditions 

The  standards  sought  are  for  performance 
and  for  the  conditions  under  which  the 
performance  must  take  place  as  noted  in 
figure  3-1-6.  The  conditions  encountered  may 
affect  the  performance  of  a  task  or  subtask. 

Conditions  can  be  the  result  of  the 
physical  environment  (e.g.,  sea  state, 
terrain,  weather),  the  military  environment 
(e.g.,  forces  assigned,  threat,  command 
relationships)  or  the  civil  environment 
(e.g.,  political,  cultural,  economic  factors 
(USMC  2007)).  Operational  conditions 
should  be  determined  and  associated 
with  Tasks  and  Subtasks  as  appropriate. 

For  example,  the  Attribute  “hit  probability” 
for  a  sniper  rifle  maps  to  the  operational 
Task  “engage  targets”  and  forms  the  basis  of 
the  performance  threshold.  The  Attribute 
“System  Ruggedness”  also  maps  to  that 
Task,  but  serves  as  the  basis  for  the  threshold 
conditions  for  achieving  hit  probability. 

The  process  of  tracing  Attributes  has  the 
unintended  consequence  of  identifying 


Figure  3-1 -6. 
Relationship 
ofThreshold 
Performance  and 
Conditions  to  COI, 
Task,  or  Subtask 
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"Orphaned"  and  Implied 
Attributes 

During  the  Attribute  tracing 
process,  it  is  possible  to  find 
an  Attribute  that  does  not 
trace  to  any  Task  or  Subtask, 
which  might  indicate  an 
"orphaned"  Attribute  having 
little  to  do  with  the  OE/OS/ 
OSur  of  the  system  and  can  be 
ignored.  However,  care  should 
be  taken  to  ensure  that  the 
orphaned  Attribute  is  not 
indicating  the  need  to  identify 
additional  Tasks  or  Subtasks. 

The  inverse  is  also  possible, 
that  is,  Tasks/Subtasks  may 
not  be  associated  with 
existing  Attributes,  which 
indicates  the  existence  of 
implied  Attributes;  these  will 
need  to  be  identified. 

Implied  Attributes  pertain  to 
capabilities  the  system  needs 
to  perform  effectively  in  the 
operational  environment 
but  are  not  already 
identified  in  the  capabilities 
documentation. 


gaps  in  the  capabilities  documents  that 
must  be  filled  for  a  successful  evaluation. 
Using  the  example  above  with  hit 
probability  and  system  ruggedness,  the 
threshold  for  hit  probability  in  operational 
conditions  is  not  clear.  The  nominal 
conditions  (70  degrees  F  ±  10  degrees) 
defined  under  hit  probability  do  not  agree 
with  system  ruggedness  conditions  (see 
table  3-1-1). 

The  apparent  disagreement  leads  to 
the  following  clarification  question: 

“What  is  the  threshold  probability  of  hit 
under  other-than-nominal  conditions?” 

In  the  process  of  deriving  evaluation 
questions  based  on  Tasks  and  Subtasks, 
the  test  team  will  find  the  need  for  standards 
that  do  not  appear  in  the  capabilities 
documentation.  Ideally  the  test  team  will 
bring  any  questions  to  the  attention  of  the 
capabilities  officer  early  in  the  acquisition 
cycle,  while  the  capabilities  documentation 
remains  in  draft.  If  the  question  is 
identified  later  in  the  acquisition  cycle,  the 
test  team  may  use  an  SME  panel,  ideally 
including  the  DC,  CD&I  Action  Officer 
for  the  program,  to  determine  preliminary 
value  for  these  standards,  potentially 
followed  by  a  Request  for  Clarification  Letter. 

If  DC,  CD&I  responds  to  the  clarification 
letter  with  the  desired  threshold  information, 
the  DC,  CD&I  values  will  be  used.  If  they  are 
unavailable,  the  test  team  will  use  the  standards 
developed  by  their  own  SME  panels. 

Attribute  Variations 

Attributes  are  defined  as  quantitative  or 
qualitative  characteristics  of  an  element 
or  its  actions  (CJCS  2005).  The  term 
Attribute  is  used  here  generically  to  refer 
to  KPPs,  KSAs,  and  other  Attributes 
of  the  system  outlined  in  capabilities 
documents.  However,  Attributes  take  many 
shapes  and  forms,  and  not  all  Attributes 
come  from  capabilities  documents.  Some 
Attributes  are  specified  by  law,  regulation, 
or  instructions.  For  example,  DODINST 
8500.2  provides  Information  Assurance 
Attributes. 


Table  3-1-1  includes  examples  of 
Attributes  from  a  single  capabilities 
document,  the  Rapid  Engagement 
Precision  Rifle  (REPR)  CDD.  The 
examples  illustrate  a  variety  of  Attributes 
ranging  from  mandatory  components  to 
field  use  parameters. 

Mapping  Attributes  to  the 
Evaluation  Framework 

Attributes  in  the  capabilities 
documentation  should  trace  to  Subtasks, 
Tasks,  and  COIs.  The  tracing  process 
supports  identification  (and  sometimes 
development)  of  standards  for  the  COIs 
and  Issues;  in  essence,  the  minimum 
acceptable  outcome  or  effect. 

The  resulting  Evaluation  Framework  links 
satisfaction  of  COIs  to  the  capabilities 
identified  in  the  JCIDS  documents  as  the 
basis  for  accepting  the  system  (CJCS  2005). 

The  tracing  process  is  also  useful  for 
identifying  the  standards  for  Task/Subtask 
performance  and  the  conditions  under 
which  Tasks/Subtasks  are  to  be  performed. 

This  process  can  help  identify  suitability 
and  survivability  standards  as  well. 

At  this  point  the  capabilities 
documentation  plays  a  prominent  role. 
From  the  materiel  developer’s  point  of 
view  the  process  of  allocating  requirements 
begins  by  assigning  top-level  system 
requirements  to  the  various  subsystems 
and  lower-level  elements  of  the  materiel 
solution. 

The  evaluator  views  the  allocation  process 
differently.  Since  evaluation  is  concerned 
with  task  accomplishment,  the  Attribute 
mapping  process  occurs  after  the  Missions, 
Tasks,  and  Subtasks  have  been  determined 
with  the  Attributes  mapped  to  the  lowest 
level  Subtasks/Tasks.  When  the  materiel 
developer  ultimately  maps  components  and 
subcomponents  to  the  Attributes  in  the 
functional  analysis  and  MCOTEA  traces 
these  same  Attributes  to  the  Tasks  and 
Subtasks,  the  link  between  the  Capabilities 
Development,  Materiel  Development,  and 
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Table  3-1-1 .  Types  of  Attributes 


Attribute  Attribute  Description,  Threshold  (T),  and  Objective  (0)  Threshold  Performance  Threshold  Condition 

Forward  Assist 

The  REPR  shall  include  a  forward  assist.  (T  =  O) 

N/A 

N/A 

Color 

All  external  and  visible  REPR  surfaces  including 
magazines  and  suppressor  shall  have  a  dull  finish 
that  is  paintable,  consistent  with  current  camouflage 
colors  and  patterns,  and  minimizes  infrared 
signatures.  (T) 

N/A 

N/A 

Rail  System 

The  REPR  shall  have  a  MIL-STD  1913  quad 
forward  rail  system  that  is  integral  to  the  upper 
receiver.  The  12,  3,  and  9  o’clock  rails  must  be 
capable  of  maintaining  sight  zeros  while  conducting 
routine  firing  combined  with  combat  movement  and 
operational  training  drills.  (T) 

Maintain  sight  zeros 
(ambiguous) 

While  conducting  routine  firing 
combined  with  combat  movement 
and  operational  training  drills 

Precision  (KPP) 

The  REPR  shall  provide  a  precision  of  fire  <1.0 
minute  of  angle  (MOA)  at  800  meters  when  fired 
from  an  accuracy  fixture  in  nominal  conditions 
unsuppressed.  (T) 

Minute  of  Angle 
(MOA)  <  1.0 

At  800  meters  when  fired  from 
an  accuracy  fixture  in  nominal 
conditions  unsuppressed 

Hit  Probability 

A  fully  trained  and  current  sniper  firing  the  REPR 
shall  achieve  8  out  of  10  hits  (80%  probability) 
within  1.0  minutes  of  angle  (MOA)  at  800  meters 
firing  10  rounds  in  10  minutes  or  less  on  an 
“NRA  Bulls-eye”  target  under  nominal  conditions. 
Nominal  conditions  are  defined  as  70  degrees  F  +10 
degrees  and  unlimited  visibility  during  daylight.  (T) 

8  out  of  10  hits  (80% 
probability)  within  1.0 
minutes  of  angle  (MOA) 

A  fully  trained  and  current  sniper 
firing  the  REPR  at  800  meters 
firing  10  rounds  in  10  minutes 
or  less  on  a  “NRA  Bulls-eye” 
target  under  nominal  conditions. 
Nominal  conditions  are  defined 
as  70  degrees  F  +  10  degrees 
and  unlimited  visibility  during 
daylight. 

Trigger  Pull 

Pull  weight  shall  not  exceed  4  pounds.  (T) 

shall  not  exceed  4 
pounds 

N/A 

Weight 

Weight  with  scope,  sling,  bipod,  suppressor,  and 
magazine  loaded  with  20  rounds  shall  be  17  pounds 
or  less.  (T) 

shall  be  17  pounds  or 
less 

Weight  with  scope,  sling,  bipod, 
suppressor,  and  magazine  loaded 
with  20  rounds 

Multiple-Target 

Engagement 

The  REPR  shall  be  capable  of  engaging  3  E-Type 
Silhouette  targets  (modified  for  MCMP  Table 

II  showing  head,  chest,  and  pelvic  girdle  scoring 
areas)  placed  10  feet  apart  with  one  shot  a  piece  in 
the  head  or  chest  scoring  area  at  500  meters  in  15 
seconds  or  less.  (T) 

15  seconds  or  less 

The  REPR  shall  be  capable  of 
engaging  3  E-Type  Silhouette 
targets  (modified  for  MCMP 
Table  II  showing  head,  chest,  and 
pelvic  girdle  scoring  areas)  placed 
10  feet  apart  with  one  shot  a  piece 
in  the  head  or  chest  scoring  area 
at  500  meters 

Ergonomic 

Enhancements 

The  REPR  shall  have  an  adjustable  stock  and 
cheek-piece  that  shall  accommodate  shooter  length 
of  pull  adjustments/optics  alignment.  The  adjustable 
stock  shall  accommodate  cheek  weld,  stock  weld, 
and  eye  relief  of  the  5th-95th  percentile  of  Marines. 
The  stock  must  not  interfere  with  the  charging 
handle  or  cycle  of  operations  of  the  weapon  in  any 
configuration.  (T) 

1. 5  th- 95  th  percentile  of 
Marines 

2.  not  interfere  with  the 
charging  handle  or  cycle 
of  operations 

1 .  Cheek  weld,  stock  weld,  and 
eye  relief 

2.  Any  weapon  configuration 

System  Ruggedness 

The  REPR  shall  perform  reliably  in  High 
Temperature-160”  F,  Low  Temperature-minus  25° 

F,  Salt  Fog,  Sand  and  Dust,  Icing/Freezing  Rain, 
and  after  immersion  in  mud  (T=0). 

shall  perform  reliably 
(ambiguous) 

High  Temperature-160”  F, 

Low  Temperature-  minus  25° 

F,  Salt  Fog,  Sand  and  Dust, 
Icing/Freezing  Rain,  and  after 
immersion  in  mud 

Engagement 

Ranges 

The  REPR  shall  be  capable  of  engaging  targets 
between  300  and  800  meters. 

shall  be  capable  of 
engaging  targets 
(ambiguous) 

between  300  and  800  meters 
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Operational 

Effectiveness 

Is  the  Operational  Effectiveness 
of  the  XXX  system  adequate  to 
achieve  a  Mission  Capability  Level 
score  of  at  least  80  out  of  1 00? 

Measure:  Mission  Capability  Level 

Threshold:  >  80 

•  Mission 

COI-X.  Can  the  operators 
using  the  XXX  system  in  a 
sniping  mission  achieve  at 
least  a  0.50  probability  of 
kill? 

Measure:  Probability  of  kill 

Threshold:  >  0.50 
*  Task 

l-X.  Can  the  operators 
using  the  XXX  system 
engage  90%  of  the 
targets? 

Measure:  Percent  of 
targets  engaged 

Threshold:  >90% 

0  Suitability 
Characteristic 
(subordinate  to  a 
Task) 

1-X.X  Does  the  XXX 
system  have  a  mean 
rounds  between 
failure  (MRBF)  of  at 
least  2,000  rounds? 

Measure:  MRBF 

Threshold:  >  2,000 

0  Subtask 

l-X.X  Is  the  trigger 
pull  less  than  or  equal 
to  4  pounds? 

Measure:  Trigger 
Pull  (pounds) 

Threshold:  <  4 


Operational  Evaluation  is  complete. 

Attributes  Mapping  Matrix 

The  Attributes  Mapping  Matrix  is  a 
working  document  that  captures  the  work 
done  to  map  Attributes  to  the  Tasks  and 
Subtasks.  This  matrix  also  accounts  for 
any  MCOTEA-derived  implied  Attribute 
and  provides  the  references  for  developing 
standards.  Given  its  potential  size,  the  matrix 
is  probably  best  used  electronically  rather 
than  on  paper.  Like  the  OTA,  however,  the 
Attribute  Mapping  Matrix  must  be  kept 
current  and  available  in  the  official  SEP  files 
and  filed  in  the  T&E  Reference  Center. 

Establish  Standards  for  Evaluation  Questions 

With  Attributes  mapped  to  the  Evaluation 
Framework  the  evaluator  can  begin  to 
establish  standards. 

Some  standards  may  align  directly  with  the 
accomplishment  of  the  Issue  or  COT  For 
example,  if  the  Issue  at  the  Task  level  is  to 
“engage  targets”  and  the  Attribute  mapped 
to  it  is  Probability  of  Hit  greater  than  or 
equal  to  0.70,  then  the  standard  and  Task 
are  directly  aligned.  The  evaluator  should  be 
aware  that  some  Tasks  and/ or  Subtasks  may 
not  have  a  standard  that  directly  speaks  to 
the  accomplishment  of  the  Task.  In  many 
cases  the  requirements  speak  to  the  critical 
technical  parameters  of  the  materiel  solution 
rather  than  the  capability  itself.  The  evaluator 
must  decide  the  nature  of  the  evaluation 


to  take  place  at  every  level  of  the 
operational  task  hierarchy  (see  sidebar). 

At  lower  levels  in  the  hierarchy, 
evaluation  by  proxy  may  be  sufficient 
to  mitigate  risk.  Evaluation  by  proxy 
does  not  directly  measure  the  ultimate 
objective.  For  example,  measuring 
the  number  of  tanks  killed  could  be  a 
proxy  for  measuring  success  in  battle. 
Evaluating  the  task  or  subtask  directly 
may  also  be  impractical,  in  which  case 
evaluation  by  proxy  is  again  acceptable. 

The  Subtask  “squeeze  trigger”  from 
figure  3-1-3  provides  a  simple  example 
of  evaluation  by  proxy.  If  the  Task  is 
for  the  operator  to  activate  the  weapon 
system  by  squeezing  the  trigger,  then 
evaluating  the  force  required  to  activate 
the  trigger  mechanism  is  an  acceptable 
way  to  indicate  the  operator’s  ability 
to  accomplish  the  Task.  In  this  case 
the  standard  for  the  critical  technical 
parameter  becomes  the  standard  for  the 
evaluation  question  for  this  Subtask. 

In  more  complicated  circumstances, 
development  of  a  standard  for  an 
evaluation  question  may  be  the  result  of 
piecing  together  multiple  requirements 
from  lower-tiered  Subtasks  to  arrive  at 
a  COI  or  higher-tiered  Task  threshold. 
The  technique  for  accomplishing  this 
may  be  an  analytic  model,  discrete 
system  event  model,  numerical  analysis, 


Delay  Time = Move  Time + Site  Prep  Time  +  Emplace  Time  +  Usage  Time  +Displace  Time 

Equation  3-1-1.  Example  of  Measures  corresponding  to  subordinate  Tasks 


Delay  Time  =  c;  ^  ^  +  c2+  x1  +  c3  +  x2 

Where 

Cj,  c  ,  c3  =  constants 
a3  =  system  speed 
x1  =  system  emplacement  time 
x2  =  system  displacement  time 

Equation  3-1  -2.  Example  of  mathematical  expression  relating  Measures  of  subordinate  Tasks 
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or  stochastic  model.  For  example, 
determining  the  standard  for  a  COI 
where  Delay  Time  is  the  Measure  would 
be  a  function  of  times  to  accomplish 
the  subordinate  Tasks.  Equation  3-1-1 
illustrates  the  Measures  for  the  Tasks 
subordinate  to  the  COI. 

Given  this  formula,  Delay  Time  can  be 
further  expressed  mathematically  using 
equation  3-1-2  where  cx,  c2,  and  c3  are 
constants  that  have  no  bearing  on  the 
system  being  evaluated;  a1  is  the  system 
speed  (threshold  >  29  mph);  x1  is  system 
emplacement  time  (threshold  <  5  min); 
and  x2  is  system  displacement  time 
(threshold  <  15  min). 

Finalize  Evaluation  Questions 

While  the  stage  is  now  set  for  the 
evaluation  questions,  the  work  is  not 
complete.  Good  evaluation  questions  will, 
when  possible,  convey  the  performance 
criterion  or  standard  that  is  applicable 
as  well  as  the  Measure  that  is  at  issue 
(Rossi,  Lipsey,  Freeman  2004).  Each 
evaluation  question  identified  to  this  point 
should  now  be  tailored  to  incorporate  the 
standard  and  Measure. 

As  seen  in  the  OE  sidebar,  each  question 
identifies  what  is  of  concern,  how 
well  it  should  be  accomplished,  and 
the  dimension  of  measure.  When  an 
evaluation  question  lacks  one  of  these 
critical  elements,  these  shortcomings 
must  be  identified  as  early  as  possible 
and  brought  to  the  capabilities  officer’s 
attention  through  a  MCOTEA  Request 
for  Clarification  Letter. 

Developing  Measures 

Measures  are  needed  to  gather  the  data 
to  satisfy  the  evaluation  questions.  The 
Measures  dictate,  at  least  in  part,  the 
data  that  needs  to  be  gathered  as  part 
of  the  test  event.  The  Measures  will  also 
be  used  later  in  the  test  design  process 
to  determine  what  factors  (also  called 
variables)  will  be  varied  and  controlled  in 
the  testing  process. 


Types  of  Measures 

The  types  of  Measures  relevant  to  system 
evaluations  are  Measures  of  Effectiveness 
(MOE),  Measures  of  Performance 
(MOP),  Measures  of  Suitability  (MOS) 
and  Measures  of  Survivability  (MOSur). 
MOEs  are  needed  to  establish  the  system’s 
military  worth  and  value,  while  MOPs 
and  MOSs  are  needed  to  design,  build, 
and  support  the  system.  MOSurs  are  used 
to  determine  how  well  the  system  and  its 
operators  can  survive  to  accomplish  their 
mission  in  a  combat  environment. 

Properties  of  Measures 

The  evaluator  must  consider  three  initial 
properties  of  MOEs,  MOPs,  MOSs, 
and  MOSurs  when  selecting  the  best 
Measures  for  evaluation.  These  properties 
of  Measures  are  reliability,  validity,  and 
sensitivity. 

♦  Reliability  is  the  extent  to  which  the 
Measure  produces  the  same  result  when 
used  repeatedly  to  measure  the  same  thing 

♦  Validity  is  the  extent  to  which  the  Measure 
succeeds  at  measuring  what  it  is  intended  to 
measure 

♦  Sensitivity  is  the  extent  to  which  the  values 
of  the  Measure  change  when  a  change 

or  difference  occurs  in  the  thing  being 
measured 

An  effective  Measure  conveys  essential 
information  without  ambiguity  or  excess 
wording,  both  of  which  detract  from 
a  clear  understanding  of  what  data  is 
required  for  test  and  evaluation.  Examples 
of  fundamental  Measures  that  focus  on 
essential  information  include  the  examples 
in  the  sidebar  to  the  right. 

Measures  of  Effectiveness 

An  MOE  is  designed  to  correspond  to  the 
accomplishment  of  mission  objectives  and 
achievement  of  desired  results.  Generally, 
only  a  small  number  of  generic  MOEs  are 
available  to  support  system  evaluations. 


Fundamental 

Measures 

•  Power 

•  Area 

•  Flow 

•  Volume 

•  Torque 

•  Pressure 

•  Angles 

•  Frequency 

•  Temperature 

•  Velocity 

•  Distance 

•  Acceleration 

•  Mass 

•  Force 

•  Energy 
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Examples  of 
MOPs 

•  Probability  of  detection 

•  Ammunition  expenditure 
rate 

•  Rounds  to  adjust 

•  Casualties  per  dose 

•  Percent  of  tasks  satisfied 

•  Time  to  adjust 

•  Range  to  detection 

•  Operator  opinion  (rating) 

•  Onload  time 

•  Offload  time 

•  Embarkation  time 

•  Fuel  consumption 

•  Radioactivity 


Examples  of 
MOSs 

Time  between 
failures 

Time  to  repair 

Maintenance  ratio 

Availability 

Time  between 

maintenance 

actions 

Time  to  perform  preventive 
maintenance 

Logistics  Down 
Time 

Time  between  unscheduled 
maintenance  actions 


Evaluation  Measures  are  typically  limited 
to 

♦  Percents  of  total  events  of  a  specific  nature 

♦  The  time  it  takes  for  a  specific  event  to 
occur 

♦  The  range  at  which  specific  events  occur 

♦  A  qualitative  assessment  of  specific  events 

Depending  on  the  Issue  (evaluation 
question),  MOEs  maybe  decomposed  into 
MOPs,  MOSs  and  MOSurs. 

Measures  of  Performance 

An  MOP  measures  a  system’s  performance 
expressed  as  speed,  payload,  range,  time- 
on-station,  frequency,  or  other  distinctly 
quantifiable  performance  features.  MOPs 
may  have  a  greater  number  of  observable 
phenomena  to  measure  than  are  available 
for  MOEs.  Observable  phenomena  for 
MOPs  include  (but  are  not  limited  to) 
those  mentioned  for  MOEs  above  plus  the 
examples  in  the  MOP  sidebar. 

Measures  of  Suitability 

An  MOS  measures  an  item’s  ability  to 
be  supported  in  its  intended  operational 
environment.  An  MOS  typically  relates  to 
readiness  or  Operational  Availability,  and 
hence  Reliability,  Maintainability,  and  the 
item’s  support  structure. 

Measures  of  Survivability 

An  MOSur  examines  the  degree  to 
which  using  the  system  in  combat  places 
the  system  itself ,  the  operators,  or  other 
systems/operators  at  risk.  For  information 
and  business  systems,  survivability  is 
interpreted  as  the  ability  of  the  system  to 
maintain  the  security,  availability,  integrity, 
authentication,  and  nonrepudiation  of  the 
system’s  data. 

Preferential  Measures 

MCOTEA  has  a  preference  for  the 
types  of  Measures  used  in  evaluations. 

In  constructing  Measures,  the  evaluator 
should  consider  a  Measure’s  scale  and  its 


alignment  with  objectives. 

Measure  Scales 

Measures  are  scaled  as  either  natural  or 
constructed.  A  natural  scale  Measure  is 
one  found  in  general  use  and  having  a 
common  interpretation:  “number  of  kills” 
is  a  natural  scale  Measure  for  lethality  of 
a  system.  Natural  scale  Measures  provide 
efficiency  for  the  evaluator  because  they  do 
not  require  scale  definition.  Their  use  may 
also  be  less  controversial  than  constructed 
Measures  because  they  are  in  general  use. 
The  difficulty  is  that  natural  scales  may  not 
fit  the  intended  use,  depending  on  what  is 
being  evaluated. 

A  constructed  scale  Measure  is  developed 
for  a  particular  problem  to  measure  the 
degree  of  attainment  of  an  objective. 
Constructed  scales  are  used  in  a  variety  of 
situations  where  natural  scale  Measures  are 
not  appropriate.  Operator  opinion  (rating) 
Measures  using  Likert  scales,  for  example, 
are  constructed  scale  Measures. 

Measure  Alignment  with  Objectives 

A  direct  Measure  measures  the  degree  of 
attainment  of  an  objective,  again  using 
the  example  “number  of  kills.”  A  proxy 
Measure  reflects  the  degree  of  attainment 
of  its  associated  objective,  but  it  does  not 
directly  measure  the  ultimate  objective.  For 
example,  measuring  the  Gross  National 
Product  is  a  proxy  for  economic  well  being. 

Measure  Clarity 

Measures  can  be  continuous  or  discrete.  A 
continuous  Measure  can  take  on  an  infinite 
number  of  values  in  an  interval  or  collection 
of  intervals;  for  example,  the  “distance 
from  target”  can  be  represented  with  an 
infinite  number  of  values,  depending  on 
the  precision  of  the  instrument  of  measure. 
Continuous  Measures  provide  more 
information  and  require  fewer  resources 
than  non-continuous  discrete  Measures. 
Discrete  Measures  may  assume  only  a  finite 
or  countably  infinite  number  of  values;  for 
example,  the  “number  of  fatalities”  can  only 
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be  an  integer  value  (e.g.,  1, 2,  3,. . 

A  discrete  Measure  with  only  two  possible 
values  is  referred  to  as  binary;  for  example, 
“Pass/Fail”  is  a  binary  Measure.  When 
binary  Measures  are  used,  larger  amounts 
of  experimental  resources  are  necessary 
to  evaluate  a  system  process.  Discrete 
(i.e.,  binary)  Measures  should  be  avoided 
whenever  possible.  Continuous  Measures 
highly  correlated  to  the  binary  response 
can  be  used  in  the  analysis,  resulting  in 
large  savings  of  experimental  resources; 
for  example,  the  “vibration  of  a  device” 
(continuous)  during  processing  can  be 
highly  related  to  whether  the  device  will  be 
“defective/non-defective”  (binary). 

Several  questions 
commonly  arise  in 
developing  evaluation 
Measures: 


Establishing  Dominant  Measures 

A  COI  or  other  Issue  (derived  from  a 
Task  or  Subtask)  may  have  one  or  more 
MOE,  MOP,  MOS,  and/or  MOSur. 

When  possible  it  is  desirable  to  develop 
a  dominant  Measure  for  each  evaluation 
question.  A  dominant  Measure  is  a  single 
Measure,  which  when  evaluated,  will 
consistently  yield  the  same  answer.  When 
more  than  one  Measure  is  needed  for  a 
COI  or  Issue,  weights  must  be  assigned  to 
the  relative  importance  of  these  competing 
MOEs  for  the  decision  maker’s  awareness. 
Any  COI  or  Issue  with  more  than  one 
evaluation  Measure  must  also  adhere  to  the 
principles  of  mutual  exclusivity  to  avoid 


Table  3-1-2.  MCOTEA  Preference  for  Evaluation  Measures 


♦  Should  a  natural 
scale,  proxy, 
continuous  Measure 
be  used,  or  should 
a  constructed  scale, 
direct,  discrete 
Measure  be 
developed? 

♦  Should  an  Issue  (evaluation  question) 
be  subdivided  into  more  detailed  sub¬ 
considerations  for  which  natural  scales 
might  exist,  or  should  a  scale  be  constructed 
to  measure  the  evaluation  consideration 
without  subdividing  it  further? 

♦  Should  a  natural  scale  that  is  precise  but 
uses  technical  jargon  be  used,  or  should 
a  constructed  but  possibly  less  precise 
scale  be  used  that  some  stakeholders  may 
understand  more  readily? 

♦  How  carefully  should  the  scale  definition 
for  a  constructed  scale  be  specified? 

♦  Can  a  continuous  Measure  be  used  to 
accurately  portray  the  effectiveness  of  the 
system  process  or  be  used  in  conjunction 
with  a  highly  correlated  binary  Measure? 

Table  3-1-2  depicts  MCOTEA’s 

preferences  for  types  of  Measures  used,  1 

being  most  preferred  and  8  being  least. 


MCOTEA  Measure 

Clarity 

Preference 

Continuous 

Discrete 

Type  of  Scale 

Direct 

Proxy 

Direct 

Proxy 

Alignment  with 
Objective 

Natural 

1 

3 

5 

7 

Constructed 

2 

4 

6 

8 

double  counting.  Said  another  way,  if  more 
than  one  evaluation  Measure  indicates 
the  degree  of  attainment  for  a  particular 
objective  (that  is,  the  evaluation  Measures 
are  redundant),  then  that  objective  will 
probably  receive  more  weight  than  was 
intended  when  the  weights  are  assigned  to 
the  various  evaluation  Measures. 

Part  III.  Evaluation  Methods 

The  ability  of  an  evaluation  result  to 
withstand  scrutiny  rests  in  its  foundation, 
the  scientific  method.  An  element  of  the 
scientific  method  is  transparency  of  process, 
and  an  evaluation  model  with  explicit 
methods  provides  that  transparency. 
Furthermore,  a  transparent  evaluation 
process  can  be  repeated  by  others  to 
confirm  findings,  and  systems  can  be 
designed  with  the  full  understanding  of 
the  expectations  that  exist  all  the  way  up 
to  the  highest  levels  (OE)  in  a  predictable 


Examples  of 
MOSurs 

Probability  of  system 
detection  by  threat 

Probability  of  system  hit 
given  detection 

Probability  of  system 
damage  given  a  hit 

Probability  of  casualties 
given  a  hit 

Probability  of  working 
countermeasures 

Reaction  time  to  threat 

Probability  of  system 
defeating  the  threat 

Probability  of  information 
systems  compromise 
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manner.  Predictability  is  important  because 
it  keeps  evaluation  expectations  from 
becoming  a  moving  target  that  is  difficult 
and  expensive  to  achieve. 

Evaluation  occurs  in  a  continuum  as  the 
system  is  developed  and  test  results  become 
available.  Early  evaluations  of  the  system 
(at  the  Issue  level)  consist  of  comparing 
the  tested  results  for  each  Issue  with  its 
accompanying  standard.  At  these  early 
stages  of  evaluation  when  aggregation 
is  not  necessary,  no 
need  exists  to  construct 
an  evaluation  model. 

Generally  speaking, 
evaluation  models  are 
necessary  when  some  form 
of  aggregation  is  required 
to  collapse  multiple 
components  into  a  single 
evaluation  answer,  as  with 
evaluations  to  determine 
OE/OS/OSur. 

Properties  of  the 
Evaluation  Model 

The  evaluation  model 
is  used  to  evaluate  the 
system’s  test  results  to 
arrive  at  the  evaluation 
conclusions,  including 
OE/OS/OSur.  The  model 
may  employ  a  variety  of  techniques  to 
aggregate  and  collapse  the  information 
across  the  dimensions  of  OE/OS/OSur 
in  a  manageable  and  understandable  way. 
Most  evaluations  will  employ  some  form 
of  screening  criteria,  analytic  model,  and 
decision  model  to  facilitate  the  system 
evaluation. 

Screening  Criteria 

A  screening  criterion  is  a  binding 
constraint  on  the  system.  The  system  must 
meet  the  screening  criterion,  the  use  of 
which  can  simplify  the  evaluation  process 
(Kirkwood  1997).  Screening  criteria  can 
reduce  the  number  of  Issues  to  only  those 
essential  for  determining  worth  or  value. 


A  system  that  fails  to  meet  minimum 
screening  criteria  should  not  proceed  to 
evaluation. 

Aggregation  Method 

Care  should  be  taken  to  aggregate  only 
when  necessary.  Aggregation  is  necessary 
when  multiple  COIs  exist  in  the  hierarchy 
(i.e.,  a  multi-mission  system).  Tasks 
and  Subtasks  can  be  evaluated  and 
reported  out  individually  as  needed,  in 
accordance  with  the  TEMP, 
to  support  engineering  and 
system  progress  reviews  and 
to  mitigate  program  risk. 
Although  some  Tasks  and 
Subtasks  may  be  evaluated 
individually  to  ensure  that  the 
system  is  ready  for  IOT,  they 
may  also  be  evaluated  under 
operational  test  conditions  with 
typical  users  and  production- 
representative  articles. 

When  a  materiel  solution 
begins  to  show  performance 
shortfalls,  tradeoff  decisions 
must  be  made.  These  decisions 
are  important,  and  aggregation 
and  importance  weighting  are 
once  again  used  to  help  resolve 
the  issues. 

Properties  Necessary  for  Aggregation 

When  an  evaluation  contains  more  than 
one  COI,  the  need  exists  to  enforce 
additional  requirements,  given  the  added 
complexity  of  the  evaluation.  One  such 
complexity  is  the  evaluator’s  ability  to 
keep  all  of  the  COIs  in  mind  at  once, 
which  is  nearly  impossible  (Clemen, 

Reilly  2001).  The  accomplishment  of  one 
objective  can  also  impede  the  progress 
of  another  (Clemen,  Reilly  2001).  The 
system  under  evaluation  may  have  one  or 
more  competing  objectives  related  to  the 
COIs.  For  example,  one  mission  for  a 
system  may  require  a  high  degree  of  off¬ 
road  mobility,  while  another  mission  may 
require  high  levels  of  ballistic  protection. 


Quick  Definitions  for 
Evaluation  Models 

Collectively  Exhaustive: 

The  evaluation  covers  every 
mission  required  of  the 
system  as  well  as  all  the 
relevant  aspects  of  suitability 
and  survivability. 

Mutual  Exclusivity: 

The  same  objective  should 
be  covered  only  once  in  the 
evaluation  hierarchy;  no 
overlap  should  occur  between 
the  COIs. 
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Since  increasing  ballistic  protection  (i.e., 
adding  the  weight  of  armor)  also  reduces 
mobility,  the  two  COIs  have  competing 
objectives.  To  address  the  complexities  of 
multiple  COIs,  they  should  be  collectively 
exhaustive,  mutually  exclusive,  operable, 
and  small  in  number  (Parnell  2007).  To 
complement  the  additional  complexities 
of  evaluating  multiple  COIs,  screening 
criteria  and  weighting  should  also  be  used. 

Collectively  Exhaustive 

An  evaluation  to  support  a  decision  is 
collectively  exhaustive  if  it  includes  all 
aspects  of  a  decision  (Clemen,  Reilly 
2001,  Kirkwood  1997,  Parnell  2007).  In 
other  words,  if  the  evaluation  covers  every 
mission  required  of  the  system  as  well 
as  all  relevant  aspects  of  suitability  and 
survivability,  then  the  evaluation  will  be 
collectively  exhaustive. 

Mutual  Exclusivity 

Mutally  exclusive  COIs  means  that  a 
given  mission  should  be  covered  only 
once  in  the  evaluation  hierarchy  (Parnell 

2007,  Kirkwood  1997).  Overlap  between 
COIs,  especially  when  they  are  weighted, 
tends  to  overemphasize  the  importance  of 
a  particular  dimension  of  the  evaluation, 
sometimes  referred  to  as  “double  counting” 
(Kirkwood  1997). 

Evaluation  Framework  Operability 

An  example  of  an  operable  hierarchy  is 
shown  in  figure  3-1-3.  When  using  multiple 
COIs  a  tendency  exists  to  continue  to 
add  evaluation  considerations  to  achieve 
completeness,  until  the  framework  becomes 
so  complex  that  any  analysis  using  the 
framework  will  be  difficult  to  conduct  and 
interpret.  In  the  quest  for  completeness, 
evaluators  must  balance  the  practical  side, 
including  cost  and  time  to  complete  the 
analysis,  within  reasonable  time  limits 
(Kirkwood  1997).  For  this  reason  the  COIs 
and  MOEs  should  be  few  in  number  (DOD 

2008,  Kirkwood  1997,  Parnell  2007).  The 
COI,  and  the  accompanying  Task/Subtask 


framework,  should  be  as  small  as  possible 
without  compromising  needed  detail.  A 
smaller  framework  can  be  communicated 
more  easily  and  requires  fewer  resources 
to  estimate  performance  across  the  various 
evaluation  Measures  (Kirkwood  1997). 

Keeping  the  evaluation  framework  as  small 
as  possible  may  seem  to  contradict  the 
previous  discussion  on  OTA.  However,  if 
the  evaluations  are  accomplished  over  time, 
then  each  phase  addresses  relevant  aspects 
of  the  framework  rather  than  attempting 
to  collapse  information  from  the  bottom  to 
the  top  in  a  single  evaluation.  In  this  sense, 
taking  the  evaluations  one  layer  at  a  time  has 
the  effect  of  making  the  evaluations  smaller 
and  more  concentrated  on  the  relevant 
characteristics  of  performance/ suitability/ 
survivability  and  keeps  the  evaluation 
focused  on  a  single  level  of  the  system. 

Evaluation  Framework  Weighting 

Finally,  multiple  COIs  vying  for  the 
evaluator’s  attention  creates  the  need  for 
weighting,  which  subjectively  assigns  relative 
importance  to  competing  COIs  according 
to  the  combat  developer’s  priorities.  In  the 
earlier  example  of  high  mobility  versus 
ballistic  protection,  the  evaluator  must  know 
which  requirement  is  more  important  and  by 
how  much.  Weighting  allows  the  evaluator 
to  pay  proper  attention  to  the  missions  in 
terms  of  the  combat  developer’s  priorities. 

Building  an  Evaluation  Model 

Several  key  steps  go  into  building  an 
evaluation  model  to  determine  OE/OS/ 

OSur.  Most  evaluations  will  employ  screening 
criteria  and  analytic  and  decision  models. 

Identifying  Screening  Criteria 

Screening  criteria  can  be  thought  of  as 
gates  that  force  evaluations  to  occur  in 
steps  as  the  system  matures  or  information 
becomes  available.  Not  using  screening 
criteria  causes  information  to  pool  for 
a  one-time  massive  evaluation,  which 
is  cumbersome  and  inefficient.  Figure 
3-1-7  illustrates  a  sample  logic  flow  for 
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employing  screening 
criteria  in  an  evaluation 
model. 

Transportability  is  a 
common  characteristic  that 
may  ultimately  become 
a  screening  requirement. 

For  example,  certain 
systems  are  required  to  be 
transportable  by  CH-53E 
helicopters.  If  there  is  no 
other  way  to  operationally 
deploy  the  system, 
this  transportability 
requirement  would 
be  termed  a  screening 
requirement  because 
inability  to  be  transported 
by  this  platform 
would  prevent  mission 
accomplishment  altogether. 

In  terms  of  evaluations, 
screening  criteria  can 
be  considered  binding 
constraints  that  can  force 
a  particular  conclusion, 
such  as  “Not  Operationally 
Suitable.”  In  the  example 
above,  if  the  system  cannot 
be  transported  by  the  CH- 
53E,  then  the  system  would  automatically 
be  evaluated  “Not  Operationally  Suitable” 
regardless  of  performance  in  other 
suitability  areas.  The  system  should 
not  proceed  to  operational  evaluation 
to  determine  OE/OS/OSur  until  this 
requirement  has  been  satisfied. 

The  failure  to  successfully  satisfy  screening 
criteria  can  be  mitigated  before  the  final 
evaluation  of  OE/OS/OSur  in  one  of  two 
ways.  First,  the  system  can  be  retested 
after  appropriate  fixes  are  in  place  to 
mitigate  shortfalls.  Second,  the  owner  of 
the  requirement  can  relieve  the  materiel 
developer  of  the  requirement  by  modifying 
or  abandoning  it  altogether. 

Issues  (both  Task/Subtask  and 
survivability/ suitability)  form  the  basis 


of  screening  criteria.  Determining  which 
Issues  will  become  screening  criteria  and 
which  decisions  to  apply  them  to  is  largely 
subjective,  although  mapping  Attributes  to 
Issues  is  valuable  in  determining  screening 
criteria.  An  Issue  mapped  to  a  KPP  or 
KSA  may  be  a  candidate  for  becoming 
a  screening  criterion.  In  addition,  Issues 
(Tasks  or  Subtasks)  that  represent  a 
critical  path  to  mission  success  may  also  be 
selected  as  screening  criteria. 

Also  subjective  in  selecting  screening  criteria 
is  the  timing  of  their  use.  Evaluations 
of  system  performance/maturity  should 
occur  over  time.  However,  early  screening 
criteria  should  be  lower-level  Issues,  at 
the  Subtask  level,  for  example,  in  the 
Evaluation  Framework.  Issues  identified 
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Not  OE,  Not  OS,  Not  OSur 


as  screening  criteria  may  prevent  systems 
from  progressing  past  early  Gate  Reviews 
or  Critical  Design  Reviews  until  their 
performance  is  satisfactory  Later,  at  the 
time  of  the  OTRB  (see  chapter  3-3), 
screening  criteria  may  be  used  to  ensure 
that  the  system  is  sufficiently  mature  for 
the  rigors  of  operational  test.  Finally,  some 
screening  criteria,  such  as  safety,  may  be 
used  at  any  stage  of  the  evaluation,  because 
the  effect  on  mission  accomplishment  may 
not  be  observable  in  an  operational  test. 

Screening  criteria  that  affect  all  COIs 
are  considered  global,  while  all  others 
are  considered  local.  Global  screening 
criteria  constrain  the  evaluation  of  OE/ 
OS  if  not  satisfied,  while  local  screening 
criteria  constrain  the  MCL  of  a  COI  if 


not  satisfied.  Thus,  it  is 
acutely  important  for 
the  system  evaluator 
to  designate  only  those 
Issues  critical  to  success 
and/or  the  decision  maker 
as  screening  criteria; 
stated  another  way,  not 
all  requirements  should 
be  treated  as  screening 
criteria. 

Local  screening  criteria 
influence  one  or  more 
but  not  all  COIs.  Like 
global  criteria,  they  are 
considered  independent 
of  affiliations  that  may 
exist  with  other  screening 
criteria.  Unlike  global 
criteria,  local  screening 
criteria  can  be  associated 
with  more  than  one 
COI,  in  which  case  the 
criterion  is  considered 
independently  for  each 
COI  and,  in  essence,  is 
evaluated  multiple  times. 
Another  difference  from 
global  criteria  is  that  a 
failed  local  criterion  affects 
only  the  applicable  COI  and  not  OE/OS. 

Issues  identified  as  screening  criteria  are 
noted  in  the  Evaluation  Framework  of 
the  SEP  and  are  ultimately  included  in 
the  TEMP.  The  logic  for  using  screening 
criteria  and  their  effect  on  the  evaluation’s 
outcome  must  be  clearly  identified, 
especially  when  combined  with  an  analytic 
model  for  evaluation. 

Constructing  the  Analytic  Model 

A  model  is  a  simplified  representation 
of  some  aspect  of  the  real  world.  Models 
provide  a  concise  description  of  the 
essential  features  of  a  complex  situation. 
Formal  analytic  models  enable  the 
evaluator  to  consider  several  variables 
simultaneously.  By  temporarily  setting 
aside  unimportant  variables,  models 


Figure  3-1-7. 

Example  of  Screening 
Criteria  used  in  an 
Evaluation  Model 
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serve  as  powerful  tools  for  studying 
interrelationships  among  important 
variables. 

Analytic  models  focus  on  the  COIs,  which 
are  evaluation  questions  used  to  ascertain 
degree  of  mission  accomplishment.  In  turn, 
the  degree  of  mission  accomplishment  (or 
effect)  depends  on  performance,  suitability, 
and  survivability,  meaning  that  an  analytic 
model  for  the  COIs  should  incorporate  all 
three  of  these  dimensions.  Incorporating 
suitability  and  survivability  parameters  into 
the  analytic  model  is  critical  to  determining 
their  relative  impact  on  effectiveness. 

Simpler  models  are  better  than  complicated 
models.  The  urge  to  overpopulate  a  model 
with  an  abundance  of  parameters  should  be 
resisted  because  many  parameters  may  have 
little  or  no  real  effect  on  the  decision. 

Under  ideal  circumstances  KPPs  and 
KSAs  would  populate  the  analytic  model, 
but  given  the  potential  lack  of  consistency 
and  specificity  within  various  capabilities 
documents, this  maybe  difficult. Top- 
level  parameters  such  as  Operational 
Availability  (OS  parameter)  and  Probability 
of  Incapacitation  (OSur  parameter)  are 
likely  candidates  for  including  in  the  model. 
Finally,  selecting  parameters  to  include  in 
the  analytic  model  always  depends  on  the 
system  being  evaluated,  as  discussed  in  the 
following  sections. 

Constructing  the  Decision  Model 

As  discussed  earlier  in  this  chapter,  the 
conclusions  for  OE/OS/OSur  are  a  direct 
result  of  normalizing  mission  results  from 
COIs  to  a  common  scale,  the  MCL.  MCL 
is  a  score  used  to  assess  how  well  Marine 
operators  using  the  system  under  test  can 
be  expected  to  fulfill  their  intended  mission 
in  a  realistic  environment.  Arriving  at 
MCL  confers  four  distinct  advantages  to 
evaluation: 

♦  Provides  a  systematic  methodology  for 
arriving  at  OE,  OS,  and  OSur  conclusion 

♦  Allows  the  aggregation  of  Measures 
using  different  units  by  converting  the 


measurement  results  to  the  dimensionless 
MCL  value  function 

♦  Provides  a  framework  for  aggregation  when 
multiple  COIs  (missions)  are  an  element  of 
the  evaluation 

♦  Normalizes  evaluation  results  to  a  common 
scale  (between  0-100),  allowing  decision 
makers  responsible  for  multiple  programs 
to  assess  capabilities  across  their  portfolio  in 
consistent  terms 

The  0-100  scale  for  MCL  is  divided  into 
three  intervals,  defined  by  scores  of  50,  80, 
and  100  (table  3-1-3). The  100-level  score 
represents  the  capability  corresponding  to 
the  system  meeting  all  the  objective  values 
of  the  parameters  in  the  COI  analytic 
models.  The  score  of  80  corresponds  to 
the  threshold  values,  while  50  corresponds 
to  the  current  capability  fielded  for  this 
mission,  if  a  current  capability  exists. 


Table  3-1-3.  Definitions  of  Mission  Capability  Level 


Mission  Capability  Level  Range 

Fully  Mission  Capable 

80 

100 

Partially  Mission  Capable 

50 

<80 

Not  Mission  Capable 

0 

<50 

The  three  intervals  are  defined  more 
specifically  as  follows: 

♦  Fully  Mission  Capable  represents  the 
highest  section  of  the  interval  where 

a  system  scores  at  least  80.  A  system 
categorized  as  Fully  Mission  Capable  means 
the  system,  in  the  mission  context,  has 
achieved  at  least  the  equivalent  of  threshold 
performance.  A  system  must  be  fully 
mission  capable  to  be  considered  OE. 

♦  Partially  Mission  Capable  represents 
the  middle  section  of  the  interval  where 
a  system  scores  at  least  50  but  less  than 
80.  A  Partially  Mission  Capable  system 
is  considered  to  be  at  least  as  good  as  the 
current  capability,  but  still  falls  short  of 
the  threshold.  This  categorical  description 
only  applies  if  a  current  mission  capability 
exists  and  can  be  quantified.  When  the 
current  capability  does  not  exist,  a  system 
may  still  score  between  50  and  80;  however, 
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the  system  is  considered  Not  Mission 
Capable  in  this  range. 

♦  Not  Mission  Capable  represents  the 
lowest  section  of  the  interval  where 
a  system  scores  less  than  50.  A  Not 
Mission  Capable  system  does  not 
improve  on  current  mission  capabilities. 
Fielding  a  system  that  scores  less  than 
50  may  still  be  justified  by  other  aspects 
of  the  system,  such  as  lower  cost  or 
overcoming  technological  obsolescence. 

The  range  for  Not  Mission  Capable  is 
expanded  from  0-<50  to  0-<80  when 
no  current  mission  capability  exists  for  the 
missions  the  system  is  designed  to  address. 

With  MCL  defined,  the  next  step  in 
building  the  Evaluation  Model  is  to 
construct  the  mathematical  functions  for 
each  COI  to  be  used  in  deriving  MCL 
results  from  the  COI  Measures.  The 
functions  can  be  curvilinear  or,  more 
commonly,  piecewise  linear,  discussed  here. 

Constructing  the  piecewise  linear  function 
is  relatively  straightforward  if  standards 
have  been  established  for  the  COIs. 

Certain  data  points  are  needed  to  construct 
the  functions,  including  the  standards  that 
represent  the  objective,  threshold,  current 
capability  values,  and  a  value  to  establish  a 
zero  point.  Equation  3-1-3  and  table  3-1-4 
constitute  an  example  of  the  COI  analytical 
model  and  the  parameter  values  needed  to 
construct  the  piecewise  linear  function. 

Using  the  parameters  from  table  3-1-4 
with  equation  3-1-3,  a  piecewise  linear 
function  can  be  constructed  using  points 
calculated  for  current  capability,  threshold 
capability,  and  objective  capability.  These 
lines  are  then  plotted  on  a  graph  where  the 
x-axis  represents  the  MOE  results  and  the 
y-axis  represents  the  MCL  scale.  Figure 
3-1-8  illustrates  the  piecewise  linear 
function  for  the  data  in  table  3-1-4. 

The  current  capability  point  will  only 
be  used  for  missions  where  a  current 
capability  exists.  An  increase  in  capability 
expressed  as  a  new  mission  area  will  not 
have  a  current  benchmark  for  comparison; 
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Where 

Pd  -  Probability  of  detection 

Pa|d=  Probability  of  electronic  attack  given  detection 

P.  -  Probability  of  jamming 

R  -  Reliability 

Ao  -  Operational  Availability 

Pmismm  “  Probability  of  mission  success 


Mission  Capability  Level  Value  Function 


therefore,  no  Partially  Mission  Capable 
category  will  exist  between  50  and  <80. 
Instead,  only  two  segments,  0  to  <80 
(representing  Not  Mission  Capable)  and 
80  to  100  (representing  Fully  Mission 
Capable)  will  appear.  Figure  3-1-9 
illustrates  the  modification  to  the  piecewise 


Table  34-4. 
Example  Value 
to  Determine 
MCL  Graph 


Equation  3-1-3. 
Example  of  an 
Analytic  Model 
Used  to  Calculate 
MCL 


Figure  3-1-8. 
Example  of 
Relationship 
Between  MOE 
Results  and  MCL 


Figure  3-1-9.  Example 
of  Relationship 
Between  MOE  Results 
and  MCL  with  No 
Current  Capability  for 
the  Mission  under 
Evaluation 
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Figure  3-1-10. 
Example  of  Weighting 
Multiple  COIs  to 
Obtain  the  0E  Result 


linear  function  when  no  existing  mission 
capability  exists. 

Weighting  Elements  of 
the  Decision  Model 

An  evaluation  with  multiple  COIs  that 
must  be  aggregated  into  a  single  overall 
answer,  such  as  OE,  needs  a  weighting 
methodology  to  balance  the  multiple 
competing  objectives.  For  proper  weighting 
to  occur,  COIs  must  be  mutually  exclusive. 
Because  a  system’s  individual  missions 
do  not  usually  depend  on  each  other, 
setting  up  mutual  exclusivity  should  not 
be  difficult.  For  example,  the  outcome  of 
a  humanitarian  mission  is  not  generally 
influenced  by  the  outcome  of  a  separate 
and  distinct  attack  mission.  However,  the 
evaluator  should  be  mindful  of  this  process 
to  prevent  inadvertent  overweighting. 

The  most  effective  time  to  establish  the 
weights,  given  their  subjective  nature,  is  when 
the  COIs  are  established,  preferably  around 
MS  A  in  the  acquisition  process.  The  weighting 
should  reflect  the  needs  of  the  Warfighter  and 
the  intent  of  the  Combat  Developer.  Early 
establishment  of  COI  weighting  is  especially 


important  from  the  materiel  developers 
standpoint  because  the  developer  will  want 
to  optimize  system  performance  for  the  most 
important  mission  type  when  tradeoffs  must 
be  made.  Establishing  weights  for  the  COIs 
later  in  the  acquisition  process  could  lead 
to  an  inappropriately  optimized  system  or 
costly  re-engineering  for  the  most  critical 
missions. 

Lastly,  if  a  weight  is  determined  late  in  the 
acquisition  cycle,  it  may  be  more  reflective 
of  the  developer’s  capabilities  than  the 
Warfighter’s  needs.  Figure  3-1-10  is  a 
sample  decision  model  that  incorporates 
weights  for  the  COIs. 

Combining  Screening  Criteria,  Analytic 
Model,  and  Decision  Model 

Once  the  three  elements  of  the  evaluation 
model  (the  screening  criteria,  the  analytic 
model,  and  the  decision  model)  have  been 
developed,  the  last  setp  is  to  assemble 
the  components,  which  establishes  the 
roadmap  for  arriving  at  conclusions  as  the 
evaluation  progresses  over  time  (fig.  3-1-7 
and  3-1-10). 


MOE  I  -  Probability  of  Kill 


OE=  ^  W=(MCL) 
i=l 

Operational 

Effectiveness 

(0-100) 
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Applying  the  MCL  Function  at 
the  Next  Level  Down 

The  previous  example  was  simplified  to 
illustrate  the  process.  For  example,  each 
COI  in  the  previous  example  consisted  of 
one  MOE.  With  COIs  being  at  mission 
level,  several  MOEs  are  likely  to  apply  to 
each  mission.  In  this  case,  the  test  team 
will  need  to  apply  the  MCL  methodology 
at  a  level  lower  than  the  COI  level. 

Assume  a  COI  (mission)  comprises 
multiple  MOEs,  in  this  illustration,  three. 
Each  MOE  will  then  have  associated 
with  it  MOPs,  MOSs,  and  MOSurs  as 
seen  in  the  equations  below.  The  equations 
illustrate  that  any  MOE  can  comprise  any 
number  of  MOPs,  MOSs,  or  MOSurs. 
Each  equation  corresponds  to  the  analytic 
model  assigned  to  each  MOE. 

MOE,  =  (MOP, )  (MOPlb)(MOS1) 
(MOSur,)... 

MOE2  =  (MOP,)(MOS,  )(MOS2b) 
(MOSur2)... 

MOE3  =  (MOP3) (MO S3) (MOSur3 ) 
(MOSurJ... 

In  this  case,  each  MOE  is  associated  with 
its  corresponding  MCL  value  function  as 
seen  in  figures  3-1-11, 12,  and  13.  Each 
graph's  inflection  points  are  determined 
as  before:  the  MOE  value  corresponding 
to  the  current  capability  values  for  its 
associated  Measures  corresponds  to  50 
on  the  vertical  scale,  to  80  for  threshold 
capability  values,  and  to  100  for  objective 
capability  values. 

The  importance  level  of  multiple  MOEs 
must  be  determined  by  weighting  each 
one  relative  to  the  other.  The  sum  of  all 
the  MOE  weights  must  equal  1.  For  this 
mission,  each  MOE  will  be  associated  with 
its  corresponding  MCL  value  (determined 
from  its  decision  model)  and  the  MCL  will 
have  the  same  weight  as  its  corresponding 
MOE;  that  is,  w,  is  associated  with  MOE1 
and  MCL,;  w2  with  MOE2  and  MCL,;  w3 
with  MOE3  and  MCL3,  etc.  Once  the  MCL 


is  found  for  each  MOE,  the  weighted  sum  of 
the  corresponding  MCLs  is  calculated: 

P  .  .  =  Zw.MCL. ;  lw,  =  1 

mission  k  k7  k 

If  this  were  the  only  mission,  the  calculated 
P  is  the  result  for  OE  where  0  <  OE  < 

mission 

50  is  not  mission  capable,  50  <  OE  <  80  is 
partially  mission  capable,  and  80  <  OE  <  100 
is  fully  mission  capable. 

In  the  case  of  multiple  missions,  each  with 
its  own  weighted  MOEs,  each  mission  is 
weighted  by  a  corresponding  mission-level 
weighting,  w.,  where  all  the  mission-level 
weights  must  also  sum  to  l.The  equation 
is  now 

OE  =  £w.(P  .  .  )  Jyv  =  1 

iv  mission' 17  1 

As  with  a  single  mission,  multiple  missions 
are  as  follows: 

0  <  OE  <  50  is  not  mission  capable, 


Figure  3-1-11. 
Value  Function  for 
MOE, 


Figure  3-1-12.  Value 
Function  for  M0E2 


Figure  3-1-13. Value 
Function  for  M0E3 
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50  <  OE  <  80  is  partially  mission  capable,  and 
80  <  OE  <  100  is  fully  mission  capable. 

The  following  numerical  example  (tables 
3-1-5  and  6)  assumes  two  missions,  the 
first  having  three  MOEs  and  the  second 
having  four.  The  MCL  values  are  derived 
from  the  corresponding  MOEs. 


Table  3-1-5.  Mission  1  Values  and  Weights 


Mission  1  (mission  weight  =  0.3) 

MCL,  =  0.86 

t 

II 

O 

L n 

MCL,  =  0.61 

w2  =  0.3 

MCL3  =  0.70 

II 

o 

k) 

Table  3-1-6.  Mission  2  Values  and  Weights 


Mission  2  (mission  weight  =  0.7) 

MCL,  =0.65 

j> 

II 

O 

MCL2  =  0.77 

J> 

II 

O 

LaJ 

MCL3  =  0.98 

w3  =  0.2 

MCL4  =  0.87 

w,  =  0.4 

4 

The  equation  for  mission  1  is 
P  .  =  Zw.MCL  =  0.5(0.86)  +  0.3(0.61) 
+  0.2(0.70) 

(P  .  .  ),  =  0.753 

The  equation  for  mission  2  is 

P  .  .  =  Zw.MCL  =  0.1(0.65)  +  0.3(0.77) 
+  0.2(0.98)  +  0.4(0.87) 

(P  .  .  ),  =  0.84 

x  mission'  2 

And  finally,  aggregating  the  weighted 
results  for  both  missions, 

OE  =  Iw.(P  .  .  ).  =  0.3(0.753)  +  0.7(0.84) 
=  0.814 

The  system  is  Operationally  Effective. 

As  shown  in  this  example,  the  MCL  value 
function  is  determined  at  only  one  level, 
in  this  case  the  level  below  mission.  After 
that,  the  appropriate  weights  are  applied 
and  the  properly  normalized  sum  is  used 
at  the  mission  level  to  determine  overall 
mission  capability. 
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Step  2:  Test  Concept,  TEMP  Input, 
and  FD /SC  Charter  Development 


MCOTEA  develops  a  test  concept  for 
each  test  event  that  examines  an  evaluation 
question.  Test  concept  development 
directly  supports  TEMP  section  3-1,  Test 
and  Evaluation  Strategy,  by  ensuring  that 
the  test  concept  addresses  the  COIs  (IOT) 
and  Task/Subtask- level  Issues  (DT  and 
Early  Operational  Assessment  (EOA)/ 
Operational  Assessment(OA)). 

Test  concept  development  is  a  MCOTEA 
working-level  effort  (not  a  formal 
test  process  product)  maintained  in 
PowerPoint®.  When  required,  such  as 
for  programs  on  DOT&E  oversight,  this 
format  can  easily  transition  to  a  brief. 

Building  on  the  SEP,  the  MCOTEA  test 
team  develops  test  concepts  by  employing 
Design  of  Experiments  (DOE)  to  ensure 
that  a  rigorous  methodology  supports  the 
development  and  analysis  of  test  results. 

Using  the  SEP  as  the  basis  and  moving 
ahead  with  new  details,  the  OTPO/TM/ 
OA  address  the  following  topics: 

1.  System  Definition 

2.  COIs  and  Measures 

3.  Trial  Process  Flow 

4.  Factors  (table  format;  shows  Constant, 
Nuisance,  Testable,  and  Limitation  Factors) 

5.  Design  Type  (including  Reliability 
estimates,  design,  sample  size,  power  analysis) 

6.  Analytic  Method  (one-sentence 
statement  of  method  type) 

7.  Time  Estimates 

8.  Key  Resources: 

♦  Test  Range  ♦ 

♦  Test  Articles  ♦ 

♦  Threats  + 

♦  Funding  for  test 

♦  Operating  forces 
personnel 

♦  Modeling  and 
simulation 


Technical  Information  about 
Test  Concept  Development 

This  section  contains  technical  information  that 
the  test  team  will  need  to  design  test  concepts. 

Identify  the  Test  Objective 

To  identify  the  test  objective,  the  test  team 
brings  forward  information  from  the  SEP 
to  ensure  that  each  evaluation  question 
and  each  Attribute  with  a  threshold  in  the 
capabilities  documentation  is  assigned  a 
test  event.  Included  with  each  evaluation 
question  from  the  SEP  are  the  Measures, 
standards,  and  conditions  associated  with 
the  question,  which  serve  as  the  impetus 
for  the  data  collection  methodology  and 
for  the  identification  of  testable  factors. 

Review  Program  Documentation 

The  test  team  must  develop  a  thorough 
knowledge  of  the  system,  the  system’s 
mission,  the  threat  to  the  system  and 
threat  tactics,  and  the  way  its  operators 
will  employ  the  system  to  accomplish  the 
mission.  The  team  gains  this  knowledge 
by  reviewing  program  documentation, 
including  capabilities  documents  (ICD/ 
CDD/CPD);  the  system’s  COE;  the 
System  Threat  Assessment;  the  CONOPS; 
the  Marine  Corps  Task  List;  unit  TTPs; 
and  any  other  relevant  document  that 
would  help  the  team  understand  the 
missions  associated  with  the  system  under  test. 

Write  a  Request  for  Clarification  Letter 

After  reviewing  the  capabilities 
documentation,  the  test  team  may  conclude 
that  clarifications  are  required  for  certain 
capabilities  or  to  determine  standards  and 
to  ensure  that  the  test  exposes  the  system 
to  necessary  conditions.  Clarifications  may 
take  the  form  of  questions  concerning 
capabilities,  standards,  or  the  conditions 
under  which  the  system  must  perform. 

The  Request  for  Clarification,  structured 


Targets 

Instrumentation 

Specific 
requirements 
for  hardware 
(tanks,  trucks,  C4 
equipment,  etc.) 


9.  Test  limitations 
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as  a  standard  naval  letter  with  enclosures, 
should  include  any  system  capability, 
standard,  or  condition  that  may  have  been 
written  ambiguously  or  that  is  missing  entirely. 

The  test  team  addresses  the  letter  to 
the  DC,  CD&I  capabilities  sponsor 
and  the  MCSC  Program  Manager’s 
representative.  In  the  Request  for 
Clarification,  MCOTEA  offers  its 
proposed  interpretation  or  asks  that  a 
clear  interpretation  be  provided  for  each 
capability  that  may  not  be  clearly  defined. 
DC,  CD&I,  in  its  response,  concurs 
or  does  not  concur  with  MCOTEA’s 
interpretation.  Where  it  does  not  concur, 
DC,  CD&I  provides  a  clarified  response 
and  other  necessary  guidance  for  those 
items.  The  test  team  sends  a  hard  copy  of 
the  Request  for  Clarification  Letter  to 
DC,  CD&I  as  well  as  an  electronic  copy 
through  e-mail,  which  allows  DC,  CD&I 
to  enter  its  responses  directly  beneath 
MCOTEA’s  questions.  The  capabilities 
documentation  must  be  carefully  reviewed 
in  an  attempt  to  include  all  MCOTEA 
questions  in  a  single  letter;  however,  if 
additional  questions  occur  after  the  first 
Request  for  Clarification  Letter  is  issued, 
additional  letters  may  be  used. 

An  additional  Request  for  Clarification  letter 
may  be  needed  to  obtain  DC,  CD&I  input  on 
MCOTEA-derived  standards  associated  with 
evaluation  question 
Measures.  All 
questions  concerning 
the  capabilities 
documentation  must 
be  satisfactorily 
answered  before  any 
testing  related  to  the 
capabilities  under 
question  is  attempted. 

Define  the  System 

The  test  team  must 
provide  a  system 
definition  for  the 
part  of  the  system 
that  applies  to  each 


test  event.  The  system  definition  defines  the 
boundaries  (what  constitutes  the  “system” 
for  the  test,  to  include  operators)  for  the 
system  under  test  as  clearly  as  possible.  In 
early  test  concept  development  this  may 
prove  challenging,  especially  if  the  materiel 
solution  has  not  yet  been  chosen. 

Descriptions  may  vary  since  some  test 
events  exercise  only  subcomponents  of 
the  system  and  are  operated  by  non-user 
representative  operators,  whereas  other  test 
events  may  exercise  the  complete  system 
with  Marines. 

Define  the  Trial 

A  trial  is  one  observation  of  the  test.  A 
collection  of  trials  is  the  sample  that  will 
be  used  to  answer  the  relevant  evaluation 
questions.  A  trial  is  based  on  process 
flow  (step-by-step  execution  of  tasks 
and  subtasks  that  are  to  occur  during  a 
trial)  and  the  conditions  under  which  the 
trial  takes  place.  Establishing  the  process 
flow  for  an  individual  test  starts  with  the 
applicable  portion  of  the  OTA  developed 
for  the  SEP.  The  process  flow  is  used 
to  elicit  cause-effect  relationships  and 
allows  the  test  team  to  estimate  resources. 
Operational  experience  is  invaluable  in 
determining  process  flow;  familiarity  with 


Figure  3-2-1. 
Sample  Trial  Flow 
Diagram 
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Figure  3-2-2. 
Example  of  a  Fishbone 
Diagram 


unit  TTPs  can  be  very  useful  in  defining  a 
trial.  Figure  3-2-1  depicts  a  process  flow  for 
a  Joint  Tactical  Airstrike  Request  (JTAR). 

By  understanding  what  needs  to  transpire 
during  the  trials,  resource  needs  such  as 
equipment,  simulators,  targets,  and  models 
and  simulations  can  be  identified.  In 
the  simple  example  of  figure  3-2-1,  it  is 
possible  to  identify  some  major  end  items 
that  will  be  needed  for  the  test. 

Identify  Cause-Effect  Relationships 

When  trials  are  intended  to  be  identical, 
they  must  be  conducted  in  precisely  the 
same  way  every  time.  Precise  replication 
gives  the  tester  a  reasonable  chance  of 
isolating  cause-effect  relationships  when 
differences  in  effects  during  the  trial  are 
noted.  The  OTPO/TM  must  ensure  that 


the  test  team  maintains  the  discipline 
needed  to  replicate  identical  trials 
throughout  the  test. 

OAs  and  OTPO/TMs,  along  with  SMEs, 
identify  cause-effect  relationships.  The 
OA  guides  the  effort,  which  begins  as  a 
brainstorming  exercise,  and  documents  the 
results.  To  help  guide  the  process  the  OA 
can  build  Ishikawa,  or  fishbone,  diagrams 
as  seen  in  figure  3-2-2. 

Another  useful  technique  involves 
replacing  the  six  Ms  of  the  fishbone 
diagram  with  diagonal  lines  representing 
each  process  step  from  the  process  flow 
diagram.  The  brainstorming  effort  may 
generate  the  same  set  of  factors,  but 
using  process  steps  has  the  potential 
to  drive  the  brainstorming  effort  in 
a  direction  that  might  otherwise  be 
overlooked  when  developing  factors. 


The  six  Ms,  depicted  by  the  diagonal  "bones" 
of  the  figure,  contain  factors  on  each  horizontal 
line.  Not  all  factors  can  be  represented  as 
discrete  values.  For  example,  temperature 
is  a  factor  that  assumes  a  range  of  values. 
Ultimately  temperature  may  be  set  to  discrete 
blocks,  but  leaving  the  factor  as  continuous 
during  the  brainstorming  effort  is  sufficient. 


Six-M  Definitions 

Manpower  -  The  causes  attributed  to  the  people  working  on  the  process;  training  would  be  placed 
here,  for  example. 

Machine  -  The  causes  due  to  the  machines  or  the  equipment  used  in  the  process. 

Method  -  The  way  the  operation  is  conducted  to  cause  the  effect;  target  distance  or  type  of  weapon 
mount  used,  for  example. 

Materials  -  The  potential  causes  due  to  the  materials  used,  such  as  the  difference  between  two 
ammo  types. 

Measurement  -  The  causes  related  to  how  the  process  is  measured,  such  as  stopwatch  or  a  ruler. 
Mother  Nature  -  The  causes  related  to  surroundings,  such  as  external  temperature  or  humidity. 
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Factors  and  Cause  and  Effect 
Relationships 

To  begin  identifying  cause-effect 
relationships,  the  OA  and  OTPO/TM 
determine  factors,  also  called  independent 
variables.  Factors  are  things  the  tester  believes 
might  affect  the  outcome  (dependent 
variable)  of  the  trial.  See  sidebar  on  the  next 
page  for  factor  definitions. 

A  simple  example  is  a  test  designed  to 
measure  the  time  to  get  to  work  (outcome). 
Incidental  to  determining  this  time,  the 
tester  also  wants  to  know  what  effect  route 
selection  (factor)  has  on  the  time  to  get 
to  work.  Factors  can  be  broken  down  into 
levels.  The  route  selection  can  be  broken 
down  into  route- 1  and  route-2  (levels).  In 
this  case,  the  test’s  objective  is  now  twofold: 

1)  determine  the  time  to  get  to  work,  and 

2)  determine  if  route  selection  has  a  cause- 
effect  relationship  on  the  time. 

Assume  in  this  hypothetical  example  that 
although  the  speed  of  travel  was  supposed 


to  be  held  constant  during  the  trials,  it 
was  not.  In  fact,  the  speed  traveled  on  one 
route  was  noticeably  faster  than  the  other 
route.  Because  the  speed  of  travel  was  not 
adequately  controlled,  the  tester  cannot 
determine  if  route  selection  actually  has  a 
cause-effect  relationship. 

Linking  Inputs,  Process  Flows, 
and  Outputs  to  Cause-Effect 
Relationships 

After  all  factors  believed  to  affect  output 
have  been  identified,  the  next  step  is  to 
select  the  method  of  control,  which  helps 
to  categorize  factors  as  nuisance,  constant, 
testable,  and  limitation. 

Figure  3-2-3  illustrates  the  linkage  of 
the  factors  to  a  trial  process  and  output 
measures.  The  fishbone  diagram  (fig.  3-2-2) 
and  input-process-output  diagram  (fig. 
3-2-3)  are  useful  in  leading  a  discussion 
of  factors  when  brainstorming,  but  are  not 
conducive  to  formal  documentation;  a  Test 
Factors  table  is  more  appropriate  in  size 
and  shape. 


Gunner  Distance  Target 
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Nuisance  Factors 

Nuisance  factors  are 
uncontrolled  sources  of 
variation  that  represent  noise 
in  the  testing  process.  The 
influence  of  nuisance  factors 
can  be  mitigated  by  averaging 
out  their  effects,  discussed 
further  in  the  section  on 
detailed  test  planning. 

Constant  Factors 

Constant  factors  are 
controllable  but  not  of  interest 
to  the  tester  because  they  do 
not  contribute  to  answering 
the  evaluation  questions. 
These  factors  are,  as  the  name 
implies,  held  constant  for  the 
duration  of  the  test  trials. 
Doing  so  helps  ensure  that 
knowledge  gained  from  the 
test  event  is  usable. 

Testable  Factors 

Testable  factors  are  used  to 
help  answer  the  evaluation 
questions.  The  threshold 
conditions  from  the  SEP  are 
usually  the  factors  identified 
as  testable  conditions.  Factors 
selected  as  testable  are 
systematically  varied  as  inputs 
to  the  test  to  determine 
their  relative  cause-effect 
relationship  to  the  output. 

Limitation  Factors 

Factors  identified  as 
limitations  are  believed  to 
affect  the  outcome,  but  for 
resource,  logistical,  or  other 
reasons  cannot  be  dealt  with 
effectively  in  the  test  design 
process.  Limitation  factors  are 
not  of  interest  to  the  tester, 
cannot  beheld  constant,  and 
cannot  be  mitigated  through 
the  test  design  process. 
Limitations  are  documented 
in  the  test  concept  process 
(TEMP,  para  3.6.3  and  in 
MCOTEA's  Test  Plan)  to  alert 
decision  makers  of  the  risk 
involved  in  accepting  the 
limitations. 


Select  Sample  Size,  Design  Type, 
and  Analysis  Method 

Generally  speaking,  sample  sizes  and  the 
number  of  trials  are  determined  based 
on  the  balancing  of  resource  constraints, 
confidence  level,  and  the  ability  to  detect 
a  desired  effect.  Sample  sizes  are  also 
determined  by  the  design  type  selected.  A 
wide  variety  of  design  types  and  analysis 
strategies  is  available.  Basic  designs  include 
full  factorial,  partial  factorial,  and  central 
composite.  Analysis  techniques  include 
Analysis  of  Variance  (AN OVA)  and 
regression.  Based  on  the  design  selected, 
resource  estimation  can  begin  in  earnest  to 
complete  the  test  concept  process  in  support 
of  informing  and  building  a  TEMP. 

The  level  of  detail  needed  to  explain  the 
selection  of  sample  size  and  the  advantages 
and  disadvantages  of  design  types  and 
analysis  methods  is  beyond  the  scope  of  this 
publication.  Analysts  should  research  these 
topics  independently,  using  a  text  on  DOEs. 

Test  Limitations 

The  test  team  will  attempt  to  evaluate  all 
missions  and  capabilities  of  the  system; 
however,  in  some  cases  areas  will  exist 
where  the  appropriate  level  of  testing  is 
not  possible.  The  SEP  provides  a  strategy 
for  completely  evaluating  a  system  in  an 
unconstrained  test  environment,  but  the 
SEP  must  also  report  any  known,  upfront 
limitations  affecting  the  evaluation  strategy. 
A  Test  Limitation  is  a  shortfall  in  OT  depth 
or  breadth  that  may  affect  the  resolution  of 
a  test  Issue.  For  example,  some  conditions 
simply  cannot  be  tested  in  peacetime,  e.g., 
open-air  nuclear  detonations  cannot  be  used 
to  operationally  test  Electromagnetic  Pulse 
hardening.  Limitations  are  also  created  by 
cost,  schedule,  or  facilities;  such  limitations 
may  not  be  acceptable  to  the  Director, 
MCOTEA  or  the  MDA,  and,  therefore, 
must  be  clearly  and  prominently  described 
in  the  TEMP. 

A  Test  Limitation  highlights  an  area  where 
the  performance  of  the  system  under  test 


may  not  be  completely  known  at  the 
completion  of  the  evaluation.  If  the  Test 
Limitation  implies  “inadequate”  OT,  the 
Director  will  request  that  the  MDA  either 
accept  the  increased  decision  risk  associated 
with  the  limitation  or  increase  OT  resources 
to  eliminate  the  limitation. 

The  test  team  must  identify  all  potential 
test  limitations  including  threat  realism, 
resource  availability,  limited  operational 
(military,  physical,  and  civil)  environments, 
limited  support  environment,  maturity  of 
tested  system,  inadaquate  M&S  support, 
safety,  etc.,  that  may  affect  the  resolution 
of  operational  Issues.  The  test  team  must 
prepare  a  Test  Limitations  Risk  Assessment 
as  a  MCOTEA  input  to  the  TEMP. 

SECNAVINST  5000.2  states  “When 
significant  test  limitations  are  identified, 
[Director,  MCOTEA  shall]  advise  the 
MDA  of  risk  associated  in  the  procurement 
decision.”  Accepted  Test  Limitations  must 
be  identified  in  the  TEMP  and  addressed 
in  the  appropriate  test  event  plan  and 
report.  After  the  test  has  been  executed, 
any  unanticipated  test  limitations  that  were 
encountered  must  be  identified  when  test 
results  are  reported. 

Preparing  TEMP  Input 

With  the  SEP  in  place  the  test  team 
continues  to  develop  test  plans  for  eventual 
use  in  test  execution.  The  SEP  provides  the 
groundwork  for  MCOTEA’s  participation 
in  developing  the  TEMP,  the  next  critical 
step  in  planning.  The  OTPO  and  TM  are 
responsible  for  developing  MCOTEA’s 
contributions  to  the  TEMP. 

TEMP  Background  and  Structure 

The  TEMP  is  the  contract  between  the 
developer,  user,  and  operational  tester 
that  documents  the  plans,  schedule,  and 
required  resources  of  the  T&E  program. 
The  MCSC/PEO  Land  Systems  Program 
Manager  is  responsible  for  producing  the 
TEMP  with  the  support  of  the  T&E 
Working-level  Integrated  Product  Team 
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(T&E  WIPT)  (DAU  2009  and  USMC 
2010).  MCOTEA  is  a  member  of  the 
T&E  WIPT  and  the  Director,  MCOTEA 
is  an  approving  official  for  the  TEMP. 

The  TEMP  is  a  dynamic  document 
published  in  support  of  MS  B  and  must 
be  reviewed  and  updated  as  required 
after  major  program  changes  and  at  each 
program  milestone.  The  TEMP  must  be 
consistent  with  the  acquisition  strategy, 
the  approved  ICD,  CDD,  or  CPD  as  well 
as  the  System  Threat  Assessment,  the 
Information  Support  Plan,  the  MCOTEA 
SEP,  and  other  relevant  documents.  Figures 
3-2-4  and  5  provide  general  information. 

♦  Part  I  of  the  TEMP  discusses  Purpose, 
Mission  Description,  and  System 
Description  and  is  the  PM’s  responsibility, 
although  as  a  member  of  the  T&E  WIPT, 
MCOTEA  may  provide  general  input. 

♦  Part  II,  also  the  PM’s  responsibility, 
requires  the  T&E  WIPT  to  ensure  that 
the  OT&E  schedule  will  support  the 
system  evaluation.  Schedule  is  critical  in 
supporting  the  engineering  and  decision¬ 
making  processes.  The  PM  is  primarily 
responsible  for  overlaying  test  and  reporting 
timelines  on  the  acquisition  timeline. 
MCOTEA’s  responsibility  is  to  participate 
in  the  development  of  the  POA&M  in  the 


TEMP  and  to  inform  the  PM  when  the 
requirements  of  testing  (and  reporting) 
cannot  realistically  meet  the  expectations  of 
the  acquisition  schedule. 

♦  Part  III,  to  which  MCOTEA  makes  a 
significant  contribution,  contains  the 
T&E  Strategy;  Evaluation  Framework; 
Operational  Test  Objectives;  DOE 
Factors,  Levels,  and  Response  Variables; 
and  Operational  Test  Limitations  (risk), 
among  other  topics.  This  section  includes 
information  about  all  planned  OT, 
including  any  EOAs,  OAs,  and  the  IOT. 
Any  use  of  Models  or  Simulations  is  also 
discussed,  including  the  specific  M&S;  the 
Verification,  Validation,  and  Accreditation 
( W&A)  plans;  and  how  the  M&S  will  be 
used  to  supplement  data  taken  during  testing. 

♦  Part  IV  is  a  Resource  Summary,  which 
requires  initial  test  planning  information 
from  MCOTEA. 

Preparing  TEMP  Part  III  Inputs 

MCOTEA  uses  information  from  the 
SEP  to  “feed”  the  TEMP,  although  the 
SEP’s  three  basic  sections  do  not  mirror 
the  contents  of  TEMP  Part  III,  seen  in 
figure  3-2-5.  In  particular,  the  TEMP 
employs  the  term  “Evaluation  Framework” 
in  a  different  sense  from  MCOTEA’s  use 
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MCOTEA  SEP 

I.  System  Description 

II.  Evaluation 
Framework 

III.  Evaluation  Models  and 
Methods 

TEMP  Part  III  Contents 

3.1  T&E  Strategy 

3.2  Evaluation 
Framework 

3.3  Developmental 
Evaluation  Approach 

3.4  Live  Fire  Evaluation 
Approach 

3.5  Certification  for  I0T&E 

3.6  Operational  Evaluation 
Approach — 

Includes  Limitations 

3.7  Other  Certifications 

3.8  Reliability  Growth 

3.9  Future  Test  and 
Evaluation 


Figure  3-2-4. 

The  SEP  and  TEMP 
Breakdown 


Figure  3-2-5. 
General  TEMP 
Information 
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of  the  term.  In  the  TEMP,  the  “Evaluation 
Framework”  is  a  top-level  view  of  the 
overall  evaluation  approach.  MCOTEA 
uses  the  term  to  mean  a  complete  hierarchy 
of  COIs,  Tasks,  Subtasks,  and  their 
associated  Issues.  Although  the  SEP  is 
critical  to  evaluation  at  MCOTEA,  it 
is  a  separate  document  whose  purpose 
is  different  from  the  TEMP;  therefore, 
portions  of  the  SEP  cannot  be  dropped 
directly  into  the  TEMP.  Further  analysis 
and  development  now  occur  to  build  the 
level  of  detail  that  the  TEMP  requires, 
beginning  with  allocation  of  COIs,  Issues, 
and  Attributes  with  thresholds  to  test  events. 

Allocating  COIs,  Issues,  and  Attributes 
with  Thresholds  to  Test  Events 

The  Evaluation  Framework  from  the 
MCOTEA  SEP  identifies  the  evaluation 
questions  that  must  be  answered  during 
the  test  program.  In  Part  III  of  the  TEMP, 
these  evaluation  questions  are  allocated  to 
a  testing  source  to  ensure  that  an  event  will 
be  available  to  generate  the  data  needed  to 
answer  the  question. 

A  test  event  may  satisfy  the  information 
needs  of  more  than  one  evaluation 
question;  conversely,  a  single  evaluation 
question  may  require  more  than  one  test 
event  to  be  answered  satisfactorily.  The  type 
of  information  needed  dictates  the  type  of 
test  event  required.  Test  events  are  of  two 

Table  3-2-1 .  Foeus  of  Eeents  blsic  ^  dfy°P“>‘ 

or  operational. 

PartHI  of  the  TEMP 
allocates  evaluation 
questions  and  the 
Attributes  with  thresholds 
in  the  capability 
documentation  to  test 
events.  In  allocating 
evaluation  questions  to  test 
events,  the  test  team  can 
begin  with  the  expectation 
that  test  events  from  DT 
through  IOT  will  typically 
address  these  Issues  and 


DT  (MCSC) 

E0A,0A  (MCOTEA) 

IOT  (MCOTEA) 

Issues  for  OS,  OSur 

Issues  at  the 

Subtask  level 

OE,  OS,  OSur 
determination 

Issues  at  Subtask 
level  and  below 

Issues  at  the  Task 
level 

Missions  at  the  COI 
level 

Attributes  with 
Thresholds 

Issues  for  OS,  OSur 

Issues  at  the  Task 
level 

Remaining  Attributes 
w/  thresholds  & 

Issues  at  Subtask  and 
Task  level  depending 
on  success  of  previous 
tests 

thresholds  as  seen  in  table  3-2-1. 

Although  the  goal  is  to  obtain  relevant  DT 
data  on  all  KPPs  and  thresholds  before 
IOT,  this  may  not  be  possible.  A  few 
thresholds  may  need  to  be  examined  during 
IOT  for  the  first  time,  or  re-examined  if 
significant  changes  to  the  system  have  been 
made  after  DT  data  was  collected  on  them. 
In  any  case,  as  a  result  of  the  integrated 
test  and  evaluation  strategy,  the  goal  is  for 
DT  data  to  address  all  of  the  thresholds  in 
the  approved  capabilities  documentation. 
This  will  allow  the  operational  test  team 
to  concentrate  on  the  Mission-Based 
Testing  approach  to  address  satisfaction 
of  COIs,  assess  system  impact  to  combat 
operations,  provide  additional  information 
on  the  system’s  operational  capabilities,  and 
determine  the  OE/OS/OSur  of  the  system. 

Summary  of  TEMP  Part  III 
Development 

Using  the  SEP  as  a  basis,  the  test  team 
allocates  COIs  and  Issues  to  test  events 
and  develops  test  concepts.  The  results  of 
this  work  are  applied  to  various  paragraphs 
of  TEMP  Part  III  in  conjunction  with 
MCOTEA’s  participation  in  the  T&E 
WIPT.  The  results  of  this  work  also  fill 
other  areas  of  MCOTEA’s  SEP,  such  as 
evaluation  methods  for  data  accumulated 
over  time  once  test  events  have  been 
assigned  in  the  TEMP. 

Preparing  TEMP  Part  IV  Inputs 

The  PM  drafts  Part  IV  to  include  all 
resources  required  for  all  types  of  T&E. 
Thus,  MCOTEA’s  test  team  must  ensure 
that  all  projected  resources  for  OT&E  are 
included  in  this  section,  seen  in  figure  3-2-6. 
The  T&E  WIPT  should  ensure  that  the 
number  of  required  LRIP  items  is  explicitly 
stated  in  Part  IV. 

Identifying  Resource  Requirements 

An  experienced  operational  expert  and  a  test 
management  professional  are  essential  in 
identifying  required  resources  for  a  successful 
test.  The  preceding  discussions  about  test 
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concept  focused  on  what  is  to  be  done;  the 
test  team  now  focuses  on  how,  where,  and 
what  is  needed  to  accomplish  the  testing. 

Identify  Test  Locations 

The  T&E  WIPT’s  job  is  to  identify  ranges, 
laboratories,  and/or  facilities  needed  for  the 
test,  based  on  the  following  top-level  factors: 

♦  What  is  to  be  accomplished  (test  concept) 

♦  Conditions  necessary  for  testing  (factors) 

♦  Timeframe  required  to  support  the 
appropriate  decisions  (test  schedule) 

♦  Test  range  capability  for  gathering  required  data 

♦  Cost  of  available  sites  (which  are  most 
cost-effective?) 

Other  considerations  include  proximity 
to  the  personnel  or  using  unit,  support 
facilities,  billeting,  and  infrastructure 
requirements  such  as  maintenance 
bays,  wash  racks,  secure  stowage,  and 
transportation  available  at  testing  locations. 

Estimate  Time  Requirements 

Estimating  the  time  requirements  for  test 
events  is  based  on  sample  size,  time  to 
complete  a  single  trial,  time  for  trial  reset, 
training  and  pretest  setup  time,  on-site 
daily  transportation  and  setup  times,  and 
posttest  teardown  time. 

Identify  MEtS  Needs 

The  T&E  WIPT  develops  a  list  of  models 
and  simulations  that  can  be  acquired 
or  developed,  VV&A’d,  and  used  in 
conjunction  with  the  testing  program  to 
support  system  evaluation.  These  models 
and  simulations  can  be  used  to  supplement 
the  data  obtained  during  testing  or  as 
tools  to  analyze  the  data  obtained  during 
testing  along  with  supplemental  data 
from  other  M&S  tools.  See  chapter  6  for 
more  information  on  M&S  selection  and 
corresponding  VY&A  requirements. 

Identify  Key  Resources 

The  number  of  test  articles  depends  on 
what  the  test  is  to  accomplish  and  the 


timeframe  for  doing  so.  Since  test  articles 
are  a  resource  that  must  be  procured,  the 
test  team  must  pay  careful  attention  to 
the  quantity  and  configuration  of  the 
test  articles.  The  threshold  conditions  for 
testing  will  dictate  the  need  for  targets, 
threats,  communications  architectures, 
support  equipment,  and  instrumentation. 

Estimate  Costs 

MCOTEA  begins  the  cost  estimation 
process  for  the  TEMP  by  conducting  a 
comprehensive  analysis  of  the  specific 
resources  needed  to  support  the 
evaluation  strategy  identified  in  the  SEP. 
This  analysis  covers  all  aspects  of  the 
OT&E  program.  The  process  begins  with 
the  identification  of  key  test  resources. 

Once  the  resource  list  is  complete, 
MCOTEA  captures  the  cost  of  testing 
by  putting  cost  estimates  to  the  resources 
identified. 

Guidance  for  developing  Part  IV  of 
the  TEMP  is  found  in  the  Defense 
Acquisition  Guidebook  (DAU  2009). 
Potential  data  sources  for  cost  estimate 
development  are  the  Program  Office, 
the  MCOTEA  S-4,  the  MCOTEA  test 
archives,  and  the  MCOTEA  Lessons 
Learned  from  other  OT  events. 

In  addition,  tools  are  available  within 
MCOTEA  to  assist  in  cost  estimation. 
Most  of  these  tools  are  spreadsheet- 
based  and  come  with  a  representative 
collection  of  resources  that  are  found  in 
the  majority  of  the  OT  situations.  All 
can  be  easily  modified  by  the  test  team  to 
reflect  a  tailored  list  of  resource  categories 
appropriate  for  a  specific  test.  MCOTEA 
develops  the  OT  funding  summary  by  fiscal 
year,  aligned  to  major  events  or  phases. 

Inputs  and  the  Operational 
Test  Plan 

Preparing  TEMP  Part  IV  input  is  the 
point  at  which  the  MCOTEA  test  team 
commences  detailed  test  planning  for 


TEMP  Part  IV 

Topics  in  Part  IV  of  particular 
interest  to  operational 
test  planning  include  the 
following: 

4.1.1  Test  Articles  (number 
and  timing) 

4.1.2  Test  Sites  and 
Instrumentation 

4.1.3  Test  Support  Equipment 

4.1.4  Threat  Representation 

4.1.5  Test  Targets  and 
Expendables 

4.1.6  Operational  Force  Test 
Support 

4.1.7  Modeling  &  Simulation 
and  Testbeds 

4.1.9  Special  Requirements 
(special  databases,  geodesy, 
physical  requirements,  etc.) 

4.3  Manpower/Personnel 
Training  (this  may  be  key  to 
successful  OT) 

4.4T&E  Funding 
Requirements  (Funding 
Schedule  by  FY,  including 
pre/post  OT  support 
requirements) 


Figure  3-2-6. 

Topics  Covered  in  TEMP 
Part  IV 
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Operational  Assessments  and  tests.  The 
details  prepared  for  TEMP  Part  IV  are 
pulled  forward  into  MCOTEA’s  Test 
Plan  for  further  development  before  test 
execution. 

FD/SC  Charter  Preparation 

An  OT’s  primary  objective  is  to  provide 
accurate  and  comprehensive  information  to 
the  MDA  regarding  a  system’s  operational 
performance.  Some  of  the  data  collected 
during  test  is  used  to  calculate  system 
Reliability  and  Availability.  This  data 
is  generally  collected  in  Test  Incident 
Reports,  which  document  various  types  of 
failures  during  test. 

Before  data  collection  occurs,  DC,  CD&I, 
MCSC,  and  MCOTEA  determine  the 
basic  categories  of  failures  and  the  basic 
definitions  of  what  constitutes  those  failures. 

The  failure  categories  are  formalized  in 
the  FD/SC  Charter,  which  establishes, 
up  front,  the  guidelines  used  to  classify 
the  cause  and  effect  of  test  incidents.  The 
outcome  of  scoring  these  incidents  is  used 
to  determine  the  Reliability  estimates  for 
that  system  at  that  point  in  time. 

A  single  FD/SC  Charter  should  be 
developed  before  testing  begins  and  used 
for  all  contractor  and  government  testing 
to  score  test  incidents  during  DT  and  OT. 
All  three  parties  sign  this  agreement  and  it 
becomes  Annex  A  of  the  SEP. 

At  a  minimum  the  FD/SC  Charter 
should  contain  the  following  information: 

♦  FD/SC  conference  membership  and 
responsibilities 

♦  Rules  of  conduct  for  the  FD/SC  conferences 

♦  System  description,  including  components 
that  are  government-furnished  and  contractor- 
fiirnishea  equipment 

♦  Mission  Essential  Functions  (mef ) 

♦  Classification,  chargeability,  hazard  severity, 
and  hazard  probability  guidelines 


Mission  Essential  Functions 

Mefs  are  the  focal  point  of  every  charter 
and  the  element  unique  to  every  system. 
Mefs  should  flow  directly  from  the  OTA. 
They  may  also  be  derived  from  capabilities 
documents  or  developed  during  the  charter 
process  with  concurrence  from  DC, 

CD&I.  The  test  team  reviews  the  system’s 
operational  mission  profiles  and  mission 
scenarios  and  develops  a  short  list  of 
functions  or  tasks  the  system  must  be  able 
to  perform  to  accomplish  its  mission.  For 
example,  a  resupply  truck  must  be  able  to 
move  to  accomplish  its  mission  of  resupply, 
and  a  radio  must  be  able  to  transmit 
digital  or  voice  signals  to  accomplish  its 
communication  mission. 

Mission  Essential  Functions  are  the 
basis  of  the  FD/SC  Scoring  Conference. 
The  failure  of  a  system  to  perform  any 
of  its  mefs  during  test  results  in  an 
Operational  Mission  Failure  (OMF),  which 
adversely  affects  the  system’s  Reliability 
and  Availability  ratings.  Therefore,  the 
development  of  mefs  is  critical  to  every  test. 

Tailoring  the  FD/SC  Charter 

Using  the  mefs  identified  in  their  analysis, 
the  test  team  tailors  the  FD/SC  Charter 
to  reflect  the  system’s  unique  elements. 
Issues  such  as  identifying  government- 
furnished  equipment  and  contractor- 
furnished  equipment  are  critical  to  the 
TIR  scoring  process. 

Developing  the  FD/SC  Charter  is  very 
much  a  collaborative  effort.  The  test  team 
should  seek  advice  from  the  Branch  Flead, 
Division  Head,  S-2  Decision  Sciences 
Lead,  the  Scientific  Advisor,  and  other 
test  unit  members  during  the  charter 
development  process. 

Incident  Classification  and 
Chargeability  Guidelines 

After  the  data  is  collected  and  usually  after 
the  OT,  all  TIRs  are  reviewed  at  the  FD/SC 
Scoring  Conference,  and  failures  are  scored 
in  accordance  with  the  rules  published 
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in  the  charter.  Chapter  3-4  contains 
information  about  the  Scoring  Conference. 

Chapter  6,  section  2  (RAM)  contains  a 
detailed  discussion  and  examples  of  the 
classification  and  chargeability  guidelines 
used  for  TIR  scoring.  Presented  here  is 
an  overview  of  the  step  by  step  scoring 
process  based  on  the  guidelines  the  charter 
contains. 

Step  1  -  Incident  Classification  (No  Test). 
The  first  step  in  the  process  for  a  given 
test  incident  is  to  determine  classification. 
In  the  classification  step,  the  members 
of  the  scoring  conference  first  determine 
if  an  incident  is  related  to  Reliability 
or  Maintainability  of  the  equipment 
as  it  will  be  expected  to  be  used  in  the 
field  environment.  Incidents  judged  not 
pertinent  to  RAM  parameters  are  classified 
as  No  Test. 

Step  2-  Incident  Classification  (Crew 
Correctable  Maintenance  Action  (CCMA) 
(Optional)).  The  second  step  in  the 
classification  process  is  to  determine  if 
the  incident  was  crew  correctable.  If  the 
incident  was  correctable  by  the  crew  within 
the  specified  time  limits  using  only  the 
system’s  onboard  tools,  repair  parts,  and 
spares,  then  the  incident  should  be  scored 
as  a  CCMA. 

Step  3  -  Incident  Classification 
(Operational  Mission  Failure).  The  third 
step  in  the  classification  process  is  to 
determine  if  the  incident  was  an  OMF.  If 
the  incident  was  a  malfunction  caused  by 
or  that  could  have  caused  the  inability  to 
perform  one  or  more  mefs,  then  it  should 
be  scored  as  an  OMF.  In  addition,  if  the 
incident  is  a  critical  or  catastrophic  hazard 
to  personnel  or  equipment,  it  should  be 
scored  as  an  OMF. 

Step  4  -  Incident  Classification  (Essential 
Maintenance  Action  (EMA)).  If  the 
incident  is  one  in  which  the  system  needed 
or  could  have  needed  corrective  action 
before  the  next  mission  could  begin,  then 
the  incident  should  be  scored  as  an  EMA. 


Step  5  -  Incident  Classification 
(Unscheduled  Maintenance  Actions 
(UMA)).  Any  incident  classified  in  steps 
2  through  4  or  any  maintenance  that  does 
not  qualify  as  a  Scheduled  Maintenance 
Action  (SMA)  is  classified  as  an  UMA. 
That  is,  for  any  maintenance  that  does  not 
qualify  as  an  SMA,  the  maintenance  must 
be  prescribed  by  an  equipment  publication. 
Furthermore,  there  must  be  enough 
latitude  in  the  time  for  the  performance  of 
the  maintenance  that  it  can  be  done  in  a 
slack  period  between  missions. 

Step  6  -  Incident  Chargeability. 

Incident  chargeability  is  the  assignment 
of  responsibility  for  the  cause  of  the 
malfunction.  Each  incident  should  be 
charged  based  on  all  information  available 
at  the  time  of  the  scoring  to  one  of  the 
following:  hardware,  software,  crew, 
maintenance  personnel,  manuals,  support 
equipment,  accidents,  or  unknown. 

Step  7  -  Hazard  Severity  Assessment. 
Hazard  Severity,  also  called  Mishap 
Severity,  is  defined  to  provide  a  qualitative 
measure  of  the  most  reasonable,  credible 
mishaps  resulting  from  personnel 
error,  environmental  conditions,  design 
inadequacies,  procedural  deficiencies,  or 
System/Subsystem  or  component  failure  or 
malfunction.  The  dollar  values  shown  should 
be  tailored  on  a  system-by-system  basis 
depending  on  the  size  of  the  system  being 
considered  to  reflect  the  level  of  concern. 

Step  8  -  Hazard  Probability  Assessment. 
Hazard  Probability,  also  called  Mishap 
Probability,  is  the  probability  that  a 
mishap  will  occur  during  the  planned  life 
expectancy  of  the  system.  It  can  be  described 
in  terms  of  potential  occurrences  per  unit  of 
time,  events,  population,  items,  or  activity. 
Assigning  a  quantitative  mishap  probability 
to  a  potential  design  or  procedural  hazard 
is  generally  not  possible  early  in  the  design 
process.  At  that  stage,  a  qualitative  mishap 
probability  may  be  derived  from  research, 
analysis,  and  evaluation  of  historical  safety 
data  from  similar  systems. 


3-2-11 


Chapter  3-2 


References 

Defense  Acquisition  University.  2009.  De¬ 
fense  Acquisition  Guidebook.  Virginia: 

Defense  Acquisition  University 
Press. 

Secretary  of  the  Navy.  2008.  Implementation 
and  Operation  of  the  Defense 
Acquisition  System  and  the 
Joint  Capabilities  Integration 
and  Development  System , 

SECNAVINST  5000.2D. 


3-2-12 


Test  Planning 


Chapter  3-3 


Step  3:  Test  Planning 


While  the  MCOTEA  test  team  prepares 
TEMP  Part  IV  input,  other  operational 
test  planning  can  and  should  occur 
simultaneously.  For  example,  the  test  team 
can  make  initial  test  site  visits,  and  the  TM 
and  OA  can  develop  test  trials. 

When  the  basic  information  required 
by  TEMP  Part  IV  is  complete,  the  SEP 
and  the  TEMP  are  aligned  and  will  only 
require  updating  as  program  elements,  such 
as  cost  and  schedule, 
continue  to  change. 

The  test  team  may  now 
turn  its  full  attention  to 
developing  the  details 
of  test  plans,  both  for 
integrated  testing  and 
operational  tests. 

The  purpose  of 
MCOTEA’s  test 
planning,  execution,  and 
reporting  activities  is 
to  prepare  for  and  conduct  individual  test 
events  in  support  of  the  overall  system 
test  and  evaluation  plan  in  the  TEMP 
and  SEP.  MCOTEA’s  involvement  with 
testing  will  vary  depending  on  the  scope 
and  size  of  the  overall  testing  program 
outlined  in  the  TEMP.  MCOTEA 
observes  and  assesses  developmental 
test  events  and  conducts  operational  test 
events,  supplemented  by  modeling  and 
simulation  as  appropriate,  to  gather  data  in 
support  of  the  system  evaluation. 

Integrated  Testing 

Integrated  Testing,  which  can  occur 
sequentially  or  simultaneously  with 
MCOTEA’s  Intermediate  or  Operational 
Assessments,  takes  place  before  IOT. 
Integrated  Testing  can  provide  MCOTEA 
with  quality  test  results  that  save 
duplication  of  effort.  MCOTEA’s  cadre 
of  test  professionals  collaboratively  plans 
and  carefully  observes  DT  events  (when 
appropriate)  and  assesses  the  quality  of 


DT  results.  MCOTEA’s  involvement  with 
testing  will  vary  depending  on  the  scope 
and  size  of  the  overall  testing  program 
outlined  in  the  TEMP. 

MCOTEA  does  not  need  to  observe 
all  developmental  test  events  set  forth 
in  the  TEMP.  For  example,  some  DT 
events  are  used  purely  for  engineering 
purposes  to  experiment  with  and  perfect 
designs,  manufacturing  techniques,  and/or 
operating  and  training 
procedures.  MCOTEA’s 
participation  is  purely 
optional  in  such  cases 
because  there  is  no 
intent  to  use  data  from 
such  events  for  system 
evaluation.  However, 
MCOTEA  may  request 
or  be  invited  to  attend, 
on  a  not-to-interfere 
basis,  purely  to  gain  a 
better  understanding  of  system  operations, 
which  will  aid  in  detailed  OT  planning. 

When  MCOTEA  does  intend  to  use  data 
from  a  particular  test  event  for  system 
evaluation,  the  OTPO/TM  will  ensure  that 
a  knowledgeable  and  independent  observer 
witnesses  the  event.  The  Test  Divisions 
within  MCOTEA  are  responsible  for 
observing  and  reporting  on  test  event 
execution  and  for  analyzing  the  resultant 
test  report  for  accuracy. 

MCOTEA  follows  the  processes  described 
in  the  USMC  Integrated  T&E  Handbook 
(USMC  2010).  This  handbook  should  be 
consulted  to  answer  any  questions  dealing 
with  Integrated  Testing. 

Writing  DT  Observation  Plans 

Ideally,  MCOTEA  will  be  collaborating 
on  the  planning  for  DT  events  well  before 
they  are  executed.  However,  at  a  minimum, 
the  MCOTEA  Test  Division  obtains  the 
test  plan  from  the  DT  organization  at 


"Integrated  Testing  is  the  collaborative 
planning  and  collaborative  execution 
of  test  phases  and  events  to  provide 
shared  data  in  support  of  independent 
analysis,  evaluation,  and  reporting 
by  all  stakeholders,  particularly  the 
developmental  (both  contractor  and 
government)  and  operational  test  and 
evaluation  communities," (Secretary  of 
Defense  2008). 
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least  15  days  before  the  test  event  (USMC 
2010).  Initial  review  should  identify  those 
parts  of  the  plan  that 

♦  support  the  independent  evaluation  by 
providing  the  needed  data 

♦  need  modification  to  support  MCOTEA’s 
evaluation 

♦  are  to  be  observed  by  the  operational  tester 

The  test  team  reviews  the  plan  and  provides 
comments  to  the  PM  before  test  event 
execution  if  they  identify  inadequacies, 
inconsistencies,  or  vague  instructions 
that  rise  to  the  level  of  affecting  the  data 
MCOTEA  requires  from  the  test  event  in 
accordance  with  the  TEMP.  MCOTEA 
may  also  make  suggestions  so  the  DT  event 
will  produce  data  that  MCOTEA  can  use. 
If  the  test  team  identifies  no  problems  with 
the  test  plan  and  has  no  suggestions  from 
the  MCOTEA  perspective,  no  requirement 
exists  for  feedback.  MCOTEA  does  not 
approve/disapprove  developmental  test 
plans.  However,  if  the  operational  testers 
identify  problems  in  the  test  plan  that 
would  invalidate  test  data  previously 
planned  in  the  TEMP  for  MCOTEA’s  use, 
MCOTEA  is  obliged  to  inform  the  PM. 
This  mandatory  notification  in  done  by 
standard  naval  letter  from  the  head  of  the 
Test  Division  to  the  Program  Manager  and 
documents  that  MCOTEA  will  be  unable 
to  accept  test  event  findings  for  use  in  the 
independent  system  evaluation  unless  the 
problems  are  corrected. 

If  no  test  plan  exists,  MCOTEA  may  still 
consider  sending  an  operational  tester  to 
observe  the  event  for  system  familiarization 
purposes.  In  no  case  will  MCOTEA 
use  data  from  an  event  without  a  plan 
for  system  evaluation  purposes.  Having 
no  plan  strongly  indicates  that  results 
will  be  highly  suspect.  Without  a  plan, 
findings  are  not  reproducible  and  cannot 
be  independently  validated,  a  basic  tenet  of 
the  scientific  process.  Figure  3-3-1  depicts 
the  Observation  Plan  template. 

When  MCOTEA  chooses  to  attend  an 


Developmental  Test  Observation  Plan 
[System  Name] 

1.  Purpose.  [State  purpose  of  document,  name 
of  event,  date  and  location  of  event.  Follow  with 
purpose  of  event  itself  and  MCOTEA’s  precise 
purpose  for  being  there.  For  early  events  such  as 
Technology  Demonstrations,  MCOTEA’s  purpose 
is  to  gather  information  that  will  aid  in  planning 
future  integrated  testing.] 

Sample: 

This  document  describes  MCOTEA’s  plan  for 
observing  the  Theater  Battle  Management  Core 
System  (TBMCS)  Maintenance  Release  2  (MR2) 
Developmental  Test  (DT)  scheduled  for  14 
February-11  March  2011  at  the  Idaho  National 
Laboratory  in  Idaho  Falls,  ID.  This  multi-Service 
DT  event,  led  by  the  46th  Test  Squadron  of 
the  U.  S.  Air  Force,  will  test  the  interoperability 
and  functionality  of  TBMCS  spiral  1.1.3  MR2 
and  evaluate  its  ability  to  meet  government 
requirements  in  preparation  for  Operational  Test 
(OT)  in  August  2011.  MCOTEA  will  observe 
test  events  from  14-26  February  to  determine  the 
extent  to  which  the  Test  Plan  is  followed  and  that 
data  collection  is  comprehensive  and  complete. 

2.  Background.  [Provide  the  problem  definition 
(capability  gap)  and  a  brief  (one  paragraph)  system 
description.] 

3.  Schedule.  [State  the  test  event  schedule  from 
the  DT  Plan,  if  available.] 

4.  Organization.  [State  the  billets  of  the 
members  of  the  observation  team  (no  names); 
who  is  conducting  the  DT  event  (contractor, 
government,  etc.);  who  else  from  the  Program 
Office  may  be  attending  the  event,  etc.] 

5.  Evaluation  Questions.  [Connect  the 
DT  event  with  Issues  from  the  SEP;  e.g.,  a 
Logistics  Demonstration  event  could  be  used 
for  a  Supportability  Issue.  Identify  the  Attribute 
thresholds  that  will  be  examined  by  the  test,  if  any. 
Cite  the  section  of  the  DT  Plan  being  referenced. 
Finish  with  statement  about  the  date  MCOTEA 
expects  to  receive  the  post-event  DT  Report.] 

6.  References.  [DT  Plan, MCOTEA’s  SEP/SAP 
(do  not  reprint)  plus  any  other  references  used 
in  the  text.  Do  not  cite  general  (background) 
references.] 


Annex  A.  Data  Collection  Forms  (can  be  simple 
tables) 

Annex  B.  Incident  Response  Plan  (use  template  in 
DT  Observation  Plan  folder) 


Fig.  3-3-1. 
Developmental  Test 
Observation  Plan 
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Fig.  3-3-2 
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Considerations 
About  MCOTEA's 
Attendance  at  Early 
DT  Events 

MCOTEA  may 
attend  a  Technology 
Demonstration  or  other 
early  event  (before 
a  CDD,  etc.)  without 
intending  to  evaluate  any 
results.  The  chief  purpose 
in  attending  early  events 
is  to  gather  information 
about  the  system  to  aid 
in  developing  future 
integrated  testing. 

When  MCOTEA  is  not 
evaluating  results 
from  an  event,  the  DT 
Observation  Report 
should  state  this  and  the 
report  should  be  kept 
on  file  for  reference.  The 
Consolidated  Review 
Board  does  not  need  to 
approve  this  report. 


event  that  lacks  a  plan,  the  Test  Division  still 
writes  an  observation  plan  and  a  report  of  its 
own,  but  only  for  internal  purposes.  While 
attending  the  event,  MCOTEA  observers 
must  not  in  any  way  indicate  that  test  event 
results  will  be  used  in  system  evaluation. 

Observing  Test  Events 

Once  the  test  event  begins,  the  Test 
Divisions  responsibility  is  to  observe  test 
event  execution  for  adherence  to  the  DT 
Plan.  Under  no  circumstances  should 
MCOTEA  personnel  interfere  with  the 
conduct  of  the  event.  MCOTEA’s  function 
is  to  observe  test  conduct,  any  deviations 
from  the  DT  Plan  (no  matter  how  minor), 
changes  to  the  system  or  its  setup,  or  other 
observations  that  would  affect  the  character 
and  validity  of  the  test  event’s  data. 

A  MCOTEA  observer  with  subject  matter 
or  operational  expertise  may  feel  the 
need  to  comment  on  system  performance 
during  DT  observation  or  in  the 
subsequent  Observation  Report.  However, 
MCOTEA’s  focus  during  DT  observation 
is  on  execution  of  the  test  event,  not  system 
performance.  The  MCOTEA  observer 
attends  the  DT  event  as  a  test  professional, 
not  a  system  SME.  MCOTEA’s  purpose  in 
focusing  on  test  event  execution  is  to  build 
a  valid  data  set  over  the  life  of  the  test 
program.  Doing  so  ultimately  contributes 
to  making  the  final  system  OTA  Evaluation 
Report  completely  defensible. 

MCOTEA  personnel  may  cite  system 
performance  observations  as  causal  factors 
when  documenting  deviations  from  the  test 
plan.  For  example,  the  observer  may  need  to 
note  that  “on  the  5th  trial  the  test  was  stopped 
because  the  system  did  not  appear  to  be 
functioning.  The  remainder  of  five  planned 
trials  was  abandoned  while  the  system  was 
inspected.”  The  intent  here  is  to  provide  the 
rationale  for  test  event  deviation,  not  to  inject 
opinion  about  system  performance  adequacy. 

If  MCOTEA  is  unable  to  attend  a  DT 
event,  MCOTEA  may  use  the  data  results 
under  the  following  circumstances: 


♦  MCOTEA  has  a  copy  of  the  test  plan 

♦  A  government  representative  (can  be  a 
contractor  representing  the  government) 
familiar  with  the  system  being  tested 
witnesses  the  test 

♦  If  the  government  representative  is  a 
contractor,  this  person  cannot  be  employed 
by,  or  subcontracted  to,  the  system  developer 

♦  The  government  representative  records 
detailed  observations  of  the  test 

♦  The  government  representative  notes  all 
deviations  from  the  test  plan 

♦  The  government  representative  notes 
all  relevant  caveats  associated  with  data 
elements 

♦  The  government  representative  is  available  to 
answer  MCOTEA’s  questions  after  the  test 

♦  MCOTEA  has  access  to  all  recorded  test 
data,  the  configuration  of  the  test  asset,  and 
the  actual  test  conditions  under  which  each 
element  of  test  data  was  obtained 

♦  MCOTEA  receives  copies  of  all  reports 
generated  by  the  DT  team 

Whether  the  test  is  witnessed  by 
MCOTEA  personnel  or  not,  MCOTEA 
may  still  use  the  DT  data  to  determine  the 
extent  to  which  thresholds  are  met  and 
may  also  use  the  DT  data  to  help  identify 
risks  and  determine  OS  and  OSur.  This 
data  may  also  require  regression  testing, 
depending  on  the  circumstances.  In  any 
case,  MCOTEA  will  use  DT  data  to 
indicate  a  system’s  progress  towards  overall 
readiness  for  OT. 

DT  Observation  Reporting 

MCOTEA  expects  to  write  two  reports 
following  a  DT  event,  an  Observation 
Report  (fig.  3-3-3)  and  an  Intermediate 
Assessment  Report  (fig.  3-3-4). 

The  OTPO/Test  Manager  write  the  DT 
Observation  Report  immediately  after 
returning  from  the  DT  event.  The  process 
assumes  that  the  OTPO/TM  have  not 
yet  received  the  expected  DT  Report, 
meaning  that  only  test  execution,  not 
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system  performance,  can  be  discussed  at 
this  point.  Observers  must  refrain  from 
commenting  on  system  performance 
in  Observation  Reports  because  many 
preliminary  conclusions  levied  at  test  sites 
are  often  later  found  to  be  erroneous. 
Without  data  results  in  hand,  conclusions 
about  system  performance  remain 
opinion,  not  fact.  More  investigation  into 
causality  is  required  than  can  usually  be 
provided  on  the  test  site. 

The  Division  Head  sends  a  copy  of  the 
Observation  Report  to  the  PM  within  10 
working  days  of  event  completion. 

DT  Report  Review  Process  and 
IAR  Preparation 

After  the  DT  event  is  complete,  the 
DT  team  typically  writes  a  Test  Report, 
which  MCOTEA  should  routinely 
receive  for  evaluation  purposes  as 
agreed  to  in  the  TEMP  (USMC  2010). 
The  MCOTEA  OTPO/TM  should 
follow  up  with  inquiries  if  the  report 
is  not  received  within  the  agreed-upon 
time.  If  MCOTEA  does  not  receive 
the  DT  Report,  or  if  it  is  not  delivered 
in  time  to  allow  independent  analysis, 
evaluation,  and  reporting  before  the  Gate 
Review,  then  MCOTEA  presents  this 
information  to  the  MDA  in  lieu  of  an 
Intermediate  Assessment  Report. 

After  receiving  the  DT  Report,  the 
observer  reviews  it  carefully  to  ensure 
that  the  findings  are  accurate  and 
consistent  with  the  documented 
observations.  The  observer’s  thought 
process  should  lead  to  one  of  three 
conclusions: 

♦  Full  Concurrence  with  DT  findings. 

All  of  the  findings/results  in  the  DT 
report  are  consistent  with  MCOTEA’s 
observations  and  are  supported  by  test 
data. 

♦  Partial  Concurrence  with  DT  findings. 
Some  but  not  all  of  the  findings/ 
results  are  consistent  with  MCOTEA’s 
observations  or  test  data.  Data  may 


DT  Observation  Report 
[System  Name] 

1.  Purpose.  [State  the  purpose  of  this  document 
(to  provide  MCOTEA’s  observations  of  DT  event 
execution)  and  the  purpose  of  the  event  itself.] 

2.  Background.  [Restate  the  system  description 
from  the  Observation  Plan.] 

3.  Scope.  [Scope  of  the  report  is  what  the  observer 
saw  of  test  conduct,  without  analysis  or  conclusion.] 

4.  Objective.  [“The  objective  of  this  report  is  to 
formally  record  MCOTEA’s  observations  of  test 
execution  from  the  event  before  receiving  the  DT 
Report.”] 

5.  Assumptions,  [if  any;  brought  forward  from 
the  SEP)  Example:  MCOTEA  assumed  that  the 
system  under  test  had  reached  a  certain  level  of 
maturation  by  the  time  of  the  event.  State  any 
issues  that  may  have  been  identified  in  previous 
testing  that  have  not  been  resolved.] 

6.  Limitations,  [of  this  report.  State  that  this 
report  cannot  evaluate  test  results  without  the  Test 
Report  itself.] 

7.  Methods.  [Method  of  observation,  such  as 
tracking  DT  Plan  test  threads,  operator  surveys, 
etc.,  or  analytical  method  of  evaluation.  Include 
the  qualitative  characteristics  of  test  conduct.] 

8.  Results,  [of  observing  test  execution.  Discuss 
by  evaluation  question  or  test  objective  if  evaluation 
questions  were  not  used.  If  deviations  from  the  DT 
Plan  occurred,  discuss  them  in  detail.] 

9.  Insights.  [Preface  any  statements  here 
with  “It  appears  that”  something  about  system 
performance  may  bear  further  watching;  statement 
must  be  nonjudgmental.  Purpose  is  to  make  the 
PM  aware  of  potential  risk  areas.  Also  highlight 
positive  areas  when  notable.] 

10.  Recommendations.  [State  only 
recommendations  for  further  or  repeat  testing 
based  on  insufficiency  of  test  planning  or 
execution;  for  example,  an  Issue  not  addressed  or  a 
threshold  not  examined.] 

11.  References.  [Cite  DT  Plan  and 
Observation  Plan;  do  not  append  the  references  to 
this  report.] 

Annex  A.  Observation  Notes  for  the  Record 
(supporting  observation  data) 


Fig.  3-3-3.  DT 
Observation 
Report  Outline 
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Fig.  3-3-4. 
Intermediate  Assessment 
Report  Outline 


Intermediate  Assessment  Report 
[System  Name] 

1.  Purpose.  This  Intermediate  Assessment 
Report  presents  MCOTEA’s  evaluation  of 
test  results  from  the  [event,  date,  location] . 

This  report  is  intended  for  the  PM  and 

MDA’s  use  [at  a  Gate  Review  or  other 
purpose] .  At  the  conclusion  of  planned 
system  testing,  MCOTEA  will  aggregate 
the  results  presented  here  with  those  of 
other  developmental  and  operational  tests 
to  determine  final  system  evaluation. 

2.  Background.  [State  problem  definition 
and  system  description.] 

3.  Scope. This  report  evaluates  test  results 
from  [test  event]  only  and  is  not  intended 
to  determine  OE/OS/Sur  or  to  be  a 
comprehensive  system  evaluation. 

4.  Objective.  This  report’s  objective  is  to 
present  unbiased  evaluation  of  test  results. 

5.  Assumptions.  [Bring  forward  from 
SAP/SEP  and  other  individual  tests  as 
applicable.] 

6.  Limitations,  [of  this  evaluation, based 
on  test  deviations  or  inherent  limits  from 
the  SAP/SEP] 

7.  Methods.  [State  the  analytical  method 
of  the  evaluation.] 

8.  Results.  [Summarize  data  results  that 
highlight  risk  areas  based  on  evaluation 
questions  examined  or  how  the  system  is 
maturing  based  on  satisfying  the  evaluation 
questions. 

9.  Insights.  [State  any  verifiable  trends 
supported  by  test  results,  positive  or 
negative,  that  the  assessment  reveals.] 

10.  Conclusions.  [State  the  overall 
summary  of  evaluation  questions  without 
repeating  data.  Do  not  introduce  any  new 
ideas  or  say  anything  not  already  discussed 
in  the  text.] 

11.  Recommendations.  [State  any 
improvements,  mitigation,  or  follow- 
on  testing  needed  for  the  system. 
Recommendations  flow  from  ideas  in 

Results,  Insights,  and  Limitations.] 

12.  References,  [as  appropriate: 

SAP/SEP;  DT  Plan;  MCOTEA  DT 
Observation  Plan;  MCOTEA  Test  Report; 
DT  Report.  Do  not  reprint  or  append  any 
references.] 

Annex  A.  Analytic  Results 

be  missing,  or  an  event  may  have  been 
mischaracterized  as  “not  tested”  when  in  fact 
it  was. 

♦  No  Concurrence  with  DT  findings:  None 
of  the  findings/results  are  consistent  with 
MCOTEA’s  observations  or  consistent  with 
the  test  data. 

Early  Operational  Test 
Planning  Activities 

Operational  testing  is  defined  as  field 
testing,  under  realistic  conditions,  of  any 
item  (or  key  component)  of  weapons, 
equipment,  or  munitions  for  the  purpose 
of  determining  the  effectiveness  and 
suitability  of  the  weapons,  equipment,  or 
munitions  for  use  in  combat  by  typical 
military  users  (DAU  2005).  The  principal 
tests  that  examine  the  Task  and  Subtask 
level  are  the  Early  Operational  Assessment 
and  Operational  Assessment.  The  principal 
tests  that  examine  the  mission  level  and 
answer  COIs  are  IOT,  FOT,  and  MOT. 

Creating  the  Feasibility 
of  Support  Message 

The  detailed  effort  associated  with  setting 
up  an  operational  test  begins  when  the 
Feasibility  of  Support  (FOS)  naval  message 
is  published,  which  should  occur  between  3 
and  6  months  before  the  test’s  Operational 
Test  Readiness  Board  (OTRB).  Written 
by  the  MCOTEA  test  division,  the  FOS 
outlines  the  general  test  plan,  personnel 
requirements,  equipment  requirements, 
facility  requirements,  logistical  support,  and 
any  shortfalls  in  support  needed  for  the  test. 

When  creating  the  FOS,  the  test  team 
should  list  all  conceivable  personnel 
requirements.  (Reducing  the  numbers 
later  is  easier  than  increasing  them.)  The 
test  team  should  clarify  requirements 
using  follow-up  calls  and  e-mails  with  the 
appropriate  COMMARFOR  and  Marine 
Expeditionary  Force  (MEF)  points  of 
contact.  If  they  do  not  respond  within 
a  reasonable  time,  the  test  team  should 
consider  going  through  the  Plans,  Policies, 
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and  Operations  (PP&O)  POC  to  establish 
contact.  Often,  higher  headquarters  must 
send  a  FOS  message  to  the  Division/ 
MLG/MAW  G3  to  determine  if  units 
are  available  to  support  test  requirements. 
Ultimately,  the  MCOTEA  test  team  must 
establish  a  POC  with  the  supporting 
unit  to  facilitate  official  contact  and 
receive  a  Direct  Liaison  Authorization 
(DIRLAUTH).  Note:  When  preparing 
the  FOS  message,  the  test  team  should 
coordinate  training  requirements  with 
the  PM.  Although  the  PM  schedules 
and  conducts  operational  test  training, 
MCOTEA  is  ultimately  responsible, 
meaning  that  the  test  team  must  ensure 
that  assets,  timeline,  and  facilities/training 
areas  are  included. 

Test  Planning  Timeline 

A  notional  operational  test  planning 
timeline  of  1  year  would  dedicate  4-6 
months  to  TEMP  development.  With 


the  release  of  the  FOS  message  6  months 
before  test,  the  test  team  continues  to 
develop  and  finalize  test  plan  details, 
leading  to  the  OTRR  30  days  before  NET. 
These  time  spans  are  rules  of  thumb  (except 
for  the  30-day  pretest  OTRR  requirement) 
and  will  vary  widely  according  to  program. 

Test  Planning  Process 

Because  TEMP  Part  IV  development  and 
detailed  operational  test  planning  overlap 
and  feed  each  other,  this  section  discusses 
how  to  develop  test  details  whether  for  the 
TEMP  or  the  Operational  Test  Plan. 

Check  Lessons  Learned  Database 

The  test  team  should  begin  any  test  plan 
by  reviewing  the  MCCLL  database 
(www.mccll.usmc.mil)  for  operational 
tests  similar  to  the  current  system.  More 
information  about  reviewing  and  logging 
Lessons  Learned  is  contained  in  chapter  5. 


Operational  Test  Guidelines 

Typical  users  shall  operate  and  maintain  the  system  or  item  under  conditions  simulating  combat  stress  and  peacetime  conditions. 

The  independent  OTAs  shall  use  production  or  production-representative  articles  for  the  dedicated  phase  of  OT  that  supports  the 
full-rate  production  decision. 

The  use  of  modeling  and  simulation  shall  be  considered  during  test  planning.  As  a  condition  for  proceeding  beyond  LRIP,  I0T&E 
shall  not  be  based  exclusively  on  modeling,  simulation,  or  an  analysis  of  system  requirements,  engineering  proposals,  design 
specifications,  or  program  documents.  The  extent  of  modeling  and  simulation  usage  in  conjunction  with  OT&E  shall  be  explained  in 
the  TEMP. 

All  hardware  and  software  alterations  that  materially  change  system  performance  (OE,  OS,  and  OSur)  shall  be  adequately  tested  and 
evaluated.  This  includes  system  upgrades  as  well  as  changes  made  to  correct  deficiencies  identified  during  T&E. 

OT&E  shall  be  conducted  before  full-rate  production  to  evaluate  OE,  OS,  and  OSur  as  required  by  10  USC  2399  for  ACAT I  and  II 
programs.  (SECNAVINST  5000.2  requires  OT&E  for  all  DON  ACAT  programs  except  ACAT  IV(M)  and  AAP) 

OTAs  shall  participate  early  in  program  development  to  provide  operational  insights  to  the  combat  developers,  Program  Office,  and 
acquisition  decision  makers. 

OT&E  shall  be  structured  to  take  maximum  advantage  of  training  and  exercise  activities  to  increase  the  realism  and  scope  of  0T  and 
to  reduce  testing  costs. 

The  use  of  system  contractors  in  the  OT&E  conducted  to  support  a  decision  to  proceed  beyond  LRIP  is  restricted  by  10  USC  2399. 
(Developing  contractors  may  participate  only  to  the  extent  that  is  planned  for  them  to  be  involved  in  the  operation,  maintenance, 
and  other  support  of  the  system  being  tested  when  it  is  deployed  in  combat.) 

A  contractor  that  has  participated  (or  is  participating)  in  the  development,  production,  or  testing  of  a  system  for  a  D0D  component 
(or  for  another  contractor  of  the  D0D)  may  not  be  involved  (in  any  way)  in  the  establishment  of  criteria  for  data  collection, 
performance  assessment,  or  evaluation  activities  for  the  OT&E.  These  limitations  do  not  apply  to  a  support  contractor  that 
participates  in  such  development,  production,  or  testing,  solely  in  testing  for  the  Federal  Government. 
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Example  of 

Schedule 

Compromise 

The  ideal  analytical 
setup  for  a  tower- 
based  sensor  test 
called  for  two 
towers  to  be  rotated 
between  events. 

From  the  TM's 
viewpoint,  however, 
the  time  and  expense 
of  moving  the  towers 
negated  any  analytic 
benefit.  The  test  team 
decided  to  hold  the 
location  of  the  two 
towers  constant  and 
rotate  the  operators 
manning  the  towers. 
The  OA  was  satisfied 
that  doing  so  would 
average  out  the 
operators' influence 
between  towers, 
thus  reducing  any 
confounding  effects. 
However,  if  the 
initial  test  concept 
is  modified,  the  test 
team  must  ensure 
that  the  OA's  ability 
to  answer  evaluation 
questions  has  not 
been  impeded. 


Update  Cost  Estimates 

TEMP  Part  IV  contains  the  Integrated 
Test  resources  and  estimates  of  funding 
requirements,  although  resource  planning 
does  not  end  with  publication  of  the  TEMP. 
Resource  planning  remains  a  critical  activity 
as  the  test  team  periodically  re-evaluates 
test  schedules  and  mission  trials,  modifying 
resource  requirements  as  required.  Resource 
changes  precipitate  cost  adjustments. 

Writing  the  Test  Plan 

Detailed  test  planning  takes  the  test 
concepts  developed  to  support  the  TEMP 
and  turns  them  into  action  plans  for  test 
execution.  The  test  team  must  write  the 
plan  in  enough  detail  to  allow  anyone  with 
appropriate  knowledge  and  skills  to  execute 
the  test,  more  than  once  if  necessary.  The 
concept  of  repeatability  is  essential  to  good 
testing,  and  repeatability  can  only  occur  if  the 
plan  was  sufficiently  detailed  in  the  first  place. 

Initial  Sections  of  the  Test  Plan 

As  explained  in  detail  in  chapter  4, 
MCOTEA  abides  by  a  single  template 
for  Test  Plans.  The  initial  sections  of  the 
template  call  for  the  information  seen 
in  figure  3-3-5.  Test  Plans  contain  a 
number  of  standard  tables  (as  found  in  the 
template)  but  few  unique  graphics  apart 
from  the  tables.  Graphics  that  do  appear 
will  generally  be  maps  or  Trial  Conduct 
diagrams  unique  to  each  program. 

Refining  the  Schedule 

At  this  stage  in  test  planning,  the  test 
team  adjusts  the  initial  schedule  from 
the  Test  Concept  to  ensure  that  the  trials 
can  be  executed  with  logistical  efficiency 
while  satisfying  the  need  to  collect  high 
quality  data.  In  refining  the  test  schedule, 
the  TM  and  the  OA  may  approach  the 
same  situation  from  two  widely  different 
viewpoints.  The  TM’s  focus  is  on  executing 
trials  logically  and  efficiently,  whereas  the 
OA  may  want  sufficient  randomization  and 
blocking  to  mitigate  confounding  effects. 
Neither  viewpoint  is  entirely  right  or  wrong, 


and  the  answer  will  most  likely  involve 
compromise  (see  sidebar). 

When  deciding  between  the  two  positions, 
the  test  team  must  always  side  with  the 
opinion  that  will  produce  the  highest  quality 
data  in  the  allotted  time.  The  rationale  is 
simple.  An  efficiently  executed  test  event  with 
insufficient  analytic  controls  will  most  likely 
result  in  information  that  does  not  adequately 
explore  the  factors  of  interest  to  the  evaluator. 
The  test  team’s  purpose  is  not  simply  to 
execute  the  test;  the  foremost  purpose  of  the 
test  is  to  gather  relevant  data. 

To  mitigate  the  effect  of  changes  to  the 
schedule,  which  are  common  and  to  be 
expected,  the  test  team  should  create 
schedules  using  generic  test  days  rather 
than  calendar  days;  for  example,  Pilot  Test 
Day-1  (PT-1)  or  Record  Test  Day-5  (RT- 
5).  In  identifying  time  as  well,  the  test  team 
should  use  generic  labeling;  for  example, 
trial  4  may  have  a  start  time  of  Test  Day 
Start  +  8  hours,  indicating  that  Trial  4  will 
begin  8  hours  after  the  start  of  the  test 
day.  Specific  mention  of  time  should  occur 
only  when  environmental  conditions  such 
as  light  levels  (e.g.,  daylight,  twilight,  or 
nighttime)  are  required. 

Operational  Test  Readiness 
Board /Review  Process 

The  purpose  of  the  OTRB/OTRR  process 
is  to  ensure  that  the  test  team  and  system 
under  test  are  ready  to  proceed  to  test. 

The  OTRR  occurs  at  least  two  times 
before  IOT,  MOT,  or  FOT.  The  pre- 
OTRR  occurs  at  least  91  days  before  NET. 
Just  after  the  pre-OTRR,  MCOTEA 
holds  the  OTRB  (90  days  before  NET). 
These  reviews  are  explained  below.  The 
second  and  primary  OTRR  occurs  30 
days  before  NET.  Note  that  the  materiel 
developer  needs  to  issue  the  Pre-OTRR 
Memorandum  no  later  than  91  days  before 
OT.  See  the  USMC  Integrated  Test  and 
Evaluation  Handbook  (2010)  for  a  detailed 
explanation  of  the  OTRB/OTRR  process. 
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[Type  of]  Test  Plan 
[System  Name] 

1.  Purpose.  [Use  language  such  as  the  following:  This  Initial  Operational  Test  (IOT)  Plan 
provides  test  execution  and  management  guidance  for  the  [system].  MCOTEA  will  use 
data  obtained  from  the  IOT,  along  with  other  data  collected  during  integrated  testing,  to 
prepare  an  Operational  Test  Agency  (OTA)  Evaluation  Report  (OER),  which  will  provide 
conclusions  concerning  the  OE,  OS,  and  OSur  of  the  [system]  based  on  the  Issues  and 
Measures  contained  in  this  plan.  The  conclusions  will  be  used  to  support  a  United  States 
Marine  Corps  Milestone  C  LRIP  decision  for  the  [system]]. 

[For  EOAs  and  OAs,  state  that  the  data  will  be  used  to  evaluate  system  progress  and  to  provide 
potential  insight  into  system  trends  or  deficiencies.  For  System  Assessments,  state  that  the  data  will  be 
used  to  examine  the  risks  and  benefits  of  the  system.] 

2.  Background.  [Provide  the  problem  definition  (capability  gap)  and  system  description.] 

3.  Schedule.  [Insert  table  that  lists  dates,  events,  locations,  and  POCs.  “Test  phases”  are 
no  longer  necessary.  The  OA  must  provide  the  Trial  Sequence  before  the  schedule  can  be 
completed.] 

4.  Organization.  [Insert  chain  of  command  graphic  here  with  narrative  as  needed,  explaining 
test  team,  local  chain  of  command,  and  other  test  support  staff.  Adjust  graphic  as  needed  for 
individual  test  organization.] 

5.  Assumptions,  [if  any,  brought  forward  from  the  SEP] 

6.  Limitations,  [of  the  test.  The  Test  Limitations  described  here  will  become  Annex  A  of  the 
Test  Report.] 

7.  Executable  Test  Plan.  [This  section  of  a  Test  Plan  displays  the  information  that  the 
test  team  needs  for  successful  test  execution.  The  first  section  presents  a  global  view  of  data 
requirements  and  test  structure  in  table  format.  The  middle  section  contains  the  test  trials 
in  narrative  form.  Following  the  narrative  is  a  more  detailed  event  schedule  for  the  Test 
Manager’s  use.  The  sample  below  illustrates  how  test  details  are  filled  in.  This  process  repeats 
itself  for  each  COI/Issue.The  Measure  of  Effectiveness  is  listed  on  the  first  page  with  its 
Issue,  while  Measures  of  Suitability  and  Performance  appear  before  the  Trial  Conduct 
section.  Note  for  Pilot  Test:  begin  Trial  Sequencing  with  “PT  1,”  for  example,  and  begin  Trial 
Conduct  narrative  with  discussion  of  Pilot  Test.] 

COM :  Can  the  XXXX  system  M-1 :  Probability  of  Identification 

identify  hostile  enemy  actions  with  at 
least  a  0.70  probability  of  success? 


Fig.  3-3-5. 

Test  Plan  Outline 
(continued  on  next 
page) 


Conducting  the  Pre-OTRR 

At  least  91  days  before  NET  but  always 
before  the  OTRB,  the  pre-OTRR  is  a 
vital  opportunity  for  the  Program  Office 
and  MCOTEA  to  examine  the  system’s 
readiness  to  proceed  to  test.  Candid 
discussions  of  system  readiness  are  essential 
for  two  reasons.  First,  waiting  until  the 
last  minute  to  cancel  an  operational  test 
creates  a  burden  on  the  operating  forces  by 


impeding  their  ability  to  plan  and  train  for 
their  normal  duties.  Second,  proceeding 
to  an  operational  test  when  the  system  is 
clearly  not  ready  is  a  waste  of  valuable  test 
resources. 

Furthermore,  conducting  an  operational 
test  on  a  system  that  is  not  ready 
exacerbates  schedule  delays  in  system 
development.  The  pre-OTRR  is  chaired 
by  the  acquisition  lead  or  designated 


3-3-11 


Chapter  3-3 


representative.  Other  attendees  include  the 
MCOTEA  Division  Head,  the  MCSC 
Product  Group  Director,  the  OTPO,  the  PM, 
and  the  DC,  CD&I  Action  Officer.  After  the 
pre-OTRR,  the  Division  Head  reports  the 
level  of  the  system  readiness  for  test  to  the 


Sample  Size  and  Test  Design 

Trials  by  Variable 
Combinations 

Full  Sun 

System  1 

Sniper 

IED 

IDF 

None 

1 

1 

2 

2 

Data  Requirements  Data  Collection  Method 

Enemy  Action  (none,  sniper,  IED  emplacement) 

Form  X:  Enemy  Action,  Time  of  Day,  System  ID,  and  Operator 

ID  will  be  preloaded  for  each  planned  trial  for  the  Data 
Collectors  at  the  start  of  each  trial 

Indirect  Fire  (mortar) 

Time  of  Day  (Full  Sun,  Dusk/Dawn,  Night) 

Data  Reduction  Data  Analysis  Method 

Filter  the  records  by  system  ID 

Categorical  factors  including  Enemy  Action,  Time  of  Day, 
and  System  ID  will  be  examined  using  Binary  Logistic 
Regression  with  alpha  set  to  0.05  to  determine  if  any  factor  is 

Remove  all  records  from  the  data  that  are  not  identified 

asOpT 

a  significant  predictor  of  success 

Reso  u  rce/  Perso  n  nel  Quantity 

C0C  (RGS,  Radio)  (Critical) 

2 

0PF0R 

25 

Trial  Sequence- System  1 

Test  Day 

Trial  # 

Illumination 

RGS  Status 

Enemy 

Action 

1 

1 

Full  Sun 

On 

IED 

1 

2 

Dark 

On 

None 

Trial  Conduct.  [SAMPLEJAt  the  beginning  of  the  trials  the  COCs  will  have 
their  RGS  monitors  turned  to  the  designated  position  in  accordance  with 
the  trial  sequence.  Just  prior  to  beginning  the  event,  the  Hostile-Rifle/Scope, 
Hostile-Mortar,  Neutral,  Neutral- Rifle,  Friendly- Rifle/Scope,  and  Friendly- 
Mortar  teams  will  be  distributed  to  their  respective  positions.  Only  the  Hostile- 
Mortar,  Neutral- Rifle,  and  Friendly-Mortar  teams  can  be  visible  to  the  towers  at 
the  beginning  of  trial  #1.  [Add  maps,  diagrams,  etc.,  as  required.] 


Annex  A.  Logistics  Summary  [comprehensive  resources  and  highly  detailed 
(hour  by  hour)  daily  master  schedule.  Identify  all  Measures  and  trials.  Use 
MS  Word,  not  Excel.] 

Annex  B.  Data  Collection  Forms 

Annex  C.  Safety  Plan  [See  Templates  on  the  shared  drive.] 


Fig.  3-3-6. 
Operational  Test 
Plan,  con't. 


Director,  MCOTEA.  In  addition,  the 
acquisition  lead  will  issue  a  pre-OTRR 
memorandum  documenting  the  expected 
state  of  system  readiness  for  IOT. 
MCOTEA  uses  this  memorandum  as 
the  basis  for  scheduling  test  support  from 
the  Operating  Forces. 

Conducting  the  OTRB 

Approximately  90  days  before  NET 
but  after  pre-OTRR,  the  MCOTEA 
Division  Head  and  PGD/PM  chair  an 
OTRB  (fig.  3-3-7).  The  purpose  of  the 
OTRB  is  to  determine  the  readiness  of 
support  packages,  instrumentation,  test 
planning,  and  test  participants  to  support 
the  OT.  System  readiness  for  test  will 
have  been  determined  at  the  pre-OTRR. 
The  OTRB  identifies  any  problems  that 
may  affect  the  start  or  proper  execution 
of  the  OT  and  makes  any  necessary 
changes  to  test  plans,  resources,  training,  or 
equipment. 

Conducting  the  OTRR 

The  OTRR  is  conducted  30  days  before 
NET.  Its  purpose  is  to  determine  if 
everything  is  ready  for  the  operational  test. 
The  OTRR  is  chaired  by  the  acquisition 
lead  or  designated  representative. 
Participants  include  the  MOIC, 
representatives  from  ASN(RDA)  and 
DOT&E  (for  ACAT  I  and  II  programs), 
MCSC  Executive  Commander,  Programs, 
and  Chief  Engineer,  DC,  CD&I,  PMO 
and  MCOTEA. 

For  OTRR,  Commander  or  Executive 
Commander,  MCSC  certifies  that  the 
system  is  safe  and  ready  for  operational 
testing,  unless  otherwise  directed  by 
ASN(RD&A)  for  programs  on  the  OSD 
T&E  Oversight  List. 

The  acquisition  lead  selects  the  OTRR 
agenda  issues  based  on  SECNAVINST 
5000.2,  a  review  of  integrated 
testing  results  and  related  program 
documentation,  including  certification 
of  equipment  to  be  safe  and  ready  for 
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Process 
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New  Equipment 
Training 
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Figure  3-3-7. 

The  OTRR  and  OTRB 
Process 


OT&E.  Agenda  items  may  be  nominated 
by  any  OTRR  attendee. 

Coordination  of  Personnel 

Marine  Officer  in  Charge  and  Test  Unit 

When  it’s  time  to  schedule  operating  forces 
to  support  the  test,  the  test  team’s  top  priority 
is  to  identify  the  MOIC  and  the  test  unit. 
Depending  on  the  current  operational  tempo, 
this  task  can  be  difficult.  The  OTPO/TM 
must  establish  a  working  relationship  as 
soon  as  (but  not  before)  DIRLAUTH  is 
received  and  the  test  unit  is  assigned.  From 
this  point  on,  the  test  team  should  include 
the  MOIC  in  all  site  visits,  scheduling 
meetings,  test  plan  discussions,  etc.,  for  the 
MOIC  to  gain  a  better  understanding  of  that 
billet’s  responsibilities,  including  working 
relationships  and  chain  of  command.  The 
MOIC  need  not  be  from  the  supporting  unit, 
but  excellent  leadership  skills  are  important. 

The  MOIC  is  responsible  for  helping 
to  execute  the  test  plan  and  report  test 
deviations  to  the  OTPO,  among  other 
duties  including  the  following: 

♦  Helping  to  coordinate  necessary  resources 
required  to  support  tests 

♦  Supervising  the  Marines  conducting  the 
events  described  in  Trial  Conduct  and 
ensuring  that  Marines  collect  data  specified 
in  Data  Requirements 

♦  Ensuring  that  the  Marines  collect  the  data 


in  accordance  with  the  Test  Plan 

♦  Maintaining  a  daily  log  that  includes 
significant  events  and  incidents  that  affect 
test  conduct,  test  events  completed,  and 
personal  observations  of  the  test  conduct 
and  system  functionality 

♦  Tracking  the  daily  review,  editing,  and 
compilation  of  all  data  collection  forms  and 
electronic  data  collection 

♦  Reviewing  TIRs  for  accuracy  and 
completeness  and  providing  preliminary 
scoring  of  TIRs  for  scoring  conference 
members 

Data  Collectors 

Data  collectors  can  generally  be  acquired/ 
recruited  from  one  of  three  sources: 
government  contractors,  active  duty 
Marines  or  soldiers  (for  Joint  tests),  or,  in 
rare  cases,  Reservists,  Sailors,  and  Airmen. 

Government  Contractors 

Generally,  government  contractors  are 
the  easiest  to  schedule  and  are  arranged 
with  a  supporting  contractor.  They  are 
usually  experienced  personnel  who  require 
very  little  training  and  can  easily  adapt  to 
unexpected  test-related  situations. 

Active  Duty  Marines/ Soldiers 

Active  duty  Marines  are  able  to  fill  dual 
roles:  they  can  be  trained  as  data  collectors 
and  they  can  function  as  alternates  if  the 
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test  requires  Marines  in  a  specific  MOS  or 
if  anyone  needs  to  leave  the  test  early. 

Reservists 

The  possibility  exists  for  scheduling 
Reservists  for  duty  during  the  test.  If  a 
Reserve  Unit  resides  in  the  area,  the  test 
team  could  contact  the  I&I  (Inspector  and 
Instructor)  to  determine  if  any  Marines  need 
or  desire  active  duty  time.  Paying  Reservists 
could  be  problematic;  the  test  team  should 
check  with  the  administrative  personnel 
associated  with  the  Reserve  Support 
Center  if  the  I&I  has  no  available  funding. 
Sometimes  discretionary  funds  are  available 
to  support  active  duty  time  for  Reservists. 

Data  Requirements 
and  Test  Structure 

With  the  basics  in  place,  the  test  team  can 
now  fill  in  the  core  of  the  Test  Plan,  which 
contains  data  requirements,  test  structure, 
test  trials,  and  a  detailed  daily  schedule  for 
OTPO/TM/MOIC  use. 

To  begin  this  section  of  the  plan,  the  test 
team  brings  forward  information  from 
the  SEP  and  TEMP.  Even  if  some  of  the 
following  information  is  not  covered  in 
the  TEMP  or  the  SEP,  the  test  team  must 
address  each  item  for  each  test: 


♦  COIs/Issues 

♦  MOE,  MOP,  and/or 
MOS 

♦  Trial  Process  Flow 

♦  Test  Factors  Table 

♦  Sample  Size 

♦  Design  Type 

♦  Analysis  Method 


Time  Estimates 
Key  Resources 
Test  Range 
Test  Articles 
Threats 
Targets 

Instrumentation 
Test  Limitations 


Each  COI/Issue  is  supported  by  the  test 
and  its  respective  Measures. 

Each  COI  is  set  up  separately  in  the  plan 
with  its  Measures. 


Insert  Sample  Size  and  Test  Design 

Using  information  developed  in  the  TEMP, 
the  test  team  inserts  sample  size  and  test 


design  information  beneath  the  COI,  as 
seen  in  the  template  sample  (fig.  3-3-5). 

The  sample  size  and  test  design  table  in 
the  template  identifies  the  trials  that  allow 
sufficient  spreading  across  the  nuisance  factors 
and  testable  factors.  In  the  example  in  figure 
3-3-5,  system  and  test  day  are  the  nuisance 
factors  while  illumination,  enemy  activity,  and 
RGS  status  are  all  testable  factors. 

Develop  Data  Requirements 

The  data  requirements  (sometimes  called 
data  elements)  are  the  individual  pieces  of 
information  needed  to  satisfy  the  Measures. 
In  addition,  there  are  data  requirements 
to  conduct  appropriate  analyses  on  the 
measured  results,  such  as  establishing 
cause-effect  relationships.  A  principal  job  of 
the  OA  is  to  develop  the  data  requirements 
to  satisfy  the  evaluation  questions. 

Data  Requirements  for  Measures 

An  example  of  a  data  requirement  for  a 
Measure  is  “time  to  set  up”  for  each  trial 
for  which  the  elapsed  setup  time  must  be 
recorded.  However,  the  OA  should  take 
care  to  consider  the  widest  possible  uses  for 
the  data.  Test  data  has  a  temporal  quality 
that  should  not  be  overlooked.  Elapsed 
time  for  the  Measure  technically  satisfies 
the  Measure,  but  valuable  data  would  be 
lost  if  “when”  the  trial  took  place  was  not 
collected.  To  capture  both  the  elapsed 
time  and  the  data’s  temporal  quality,  the 
following  data  requirement  would  be 
needed  to  satisfy  the  MOP: 

♦  Time  Start  (hh:mm:ss  dd/mm/yyyy) 

♦  Time  Stop  (hh:mm:ss  dd/mm/yyyy) 

Given  this  data,  the  OA  can  use  data 
reduction  methods  (Time  Stop  -  Time 
Start  =  Elapsed  Time)  to  reduce  the  data  to 
its  usable  form  and  retain  temporal  quality 
by  knowing  when  during  the  test  period  the 
trial  took  place.  This  is  especially  important 
in  operational  tests  where  time  of  day  or 
task  sequence  is  important  in  understanding 
what  transpired  during  a  mission. 
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Reliability,  Availability,  and 
Maintainability  Data 

RAM  data  is  gathered  on  the  system 
under  test  throughout  integrated  testing; 
however,  operational  testing  is  an  excellent 
opportunity  to  gather  additional  RAM 
data.  See  chapter  6  for  more  information 
on  gathering  RAM  data  during  test. 

Determine  Data  Requirements 
for  Analysis 

For  analysis  methods  to  be  completed, 
additional  data  elements  are  required. 

Data  from  each  trial  must  be  collected  to 
support  each  factor  (constants,  nuisance, 
and  testable).  Using  the  MOP  “probability 
of  detection”  as  an  example,  the  data 
requirements  in  table  3-3-1  would  be 
needed  to  satisfy  some  of  the  factors  that 
need  to  be  analyzed. 


Develop  Data 
Collection  Methods 

Data  collection  methods  fall  into  two 
categories,  automated  and  manual,  each 
with  advantages  and  disadvantages. 

The  best  data  collection  methods  for 
operational  tests  do  not  interfere  with  the 
accomplishment  of  tasks  or  missions,  or  do 
so  to  the  slightest  possible  extent. 

Automated  Data  Collection 

Automated  data  collection  involves  some 
form  of  instrumentation  that  is  set  to 
monitor  and  record  what  is  occurring 
on  the  test  site.  Instrumentation  can  be 
installed  directly  onto  or  into  the  system 
under  test  or  on  the  test  range.  Automated 


data  collection  methods  are  useful 
when  space  requirements  limit  access  to 
personnel  outside  of  the  crew  or  operators. 
Automated  data  collection  often  can  record 
information  that  would  not  be  available 
using  manual  collection,  and  with  speed 
and  accuracy  that  manual  efforts  cannot 
duplicate. 

A  disadvantage  of  automated  data  collection 
is  that  it  typically  requires  personnel  with 
specialized  skills  to  set  up  and  operate. 
When  ranges  are  instrumented  with 
automated  data  collection  methods, 
considerable  preparation  time  may  be 
required  to  set  up  the  range  before  a  trial 
can  begin.  Additionally,  not  all  ranges  are 
suitable  for  automated  data  collection,  e.g., 
the  need  for  external  power  sources  may 
limit  automated  collection  utility  in  a  free- 
flowing  operational  event.  When  automated 
data  collection  methods  are  installed 

onboard  or  incorporated  into 
the  system  under  test,  particular 
attention  must  be  paid  to  ensure 
that  the  device  does  not  interfere 
with  system  operations. 

Manual  Data  Collection 

The  primary  focus  of  a  manual 
data  collector  is  to  observe  and 
record.  Data  collection  must 
be  limited  to  collecting  the  necessary 
data  elements,  not  scoring,  tabulating,  or 
calculating  results,  which  are  data  reduction 
functions  performed  by  the  OA.  Manual 
data  collection  can  employ  paper  or 
electronic  forms  and  has  the  advantage  of 
being  highly  adaptable  to  changes. 

However,  manual  data  collection  has 
many  disadvantages,  chief  among 
them  the  possibility  of  distractions  and 
documentation  errors;  space  requirements; 
and  training  requirements.  Other 
disadvantages  include  the  following: 

♦  Documenting  data  elements  on  a  test 

requires  attention  to  detail  and  the  ability  to 
ignore  activities  not  of  principal  concern.  A 
manual  data  collector  can  also  misinterpret 


Table  3-3-1.  Data  Requirement  Examples 


Factor  Type  Data  Element 

Nuisance 

Data  Collector 

Nuisance 

Test  Day 

Constant 

Sniper  Team  Size  (2  persons) 

Constant 

System  Location 

Testable 

Enemy  Action  (e.g.,  emplace  improvised  explosive 
device,  sniper  attack,  indirect  fire  attack) 

Definition 

Review 

Trial  =  one  test 
event  for  the  system 
under  test 

Sample  =  collection 
of  trials  within  a  test 

Process  Flow  =  step- 
by-step  depiction  of 
a  trial.  A  system  with 
multiple  missions 
(i.e.,  multiple  COIs) 
will  have  multiple 
process  flows  unless 
the  actions  from 
mission  to  mission 
do  not  vary. 
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or  incorrectly  document  an  observation.  Both 
problems  result  in  poor  quality  test  data. 

♦  Manual  data  collection  can  be  particularly 
challenging  in  tests  where  limited  space 

is  available  for  personnel  outside  of  the 
crew  or  operating  personnel.  Manual  data 
collection,  in  most  cases,  assumes  that  the 
data  collector  must  be  in  place  to  observe 
and  document  the  data  elements,  which 
requires  sufficient  space  for  that  person.  In 
tests  of  vehicle  systems  this  challenge  is 
particularly  difficult  to  overcome.  The  initial 
thought  might  be  to  use  a  crewmember  as 
a  data  collector  or  replace  a  crewmember 
with  a  data  collector,  but  neither  option 
is  viable  because  both  have  the  potential 
for  producing  a  poor  quality  test  result. 
Using  a  crewmember  as  a  data  collector 
means  tasking  that  person  with  a  job 
outside  his  particular  skill  set  that  he  may 
not  be  equipped  to  handle.  Tasking  an 
operator  with  collecting  data  may  also 
overburden  the  operator  who  retains  crew 
responsibilities.  Replacing  an  operator  with 
a  data  collector  has  similar  implications.  A 
data  collector  cannot  perform  the  duties 
of  the  replaced  operator,  and  in  any  case 
the  data  collector  must  not  be  involved  in 
system  operation;  the  data  collector  must 
remain  a  passive,  non-interfering  observer 
of  events. 

♦  One  of  the  greatest  challenges  for  a 
manual  data  collector  is  collecting  data  on 
a  multifunctional  system.  A  data  collector 
may  be  charged  with  recording  failure 
information  on  one  system  function  while 
simultaneously  recording  events  of  other 
system  functions.  Many  people  do  not 
multitask  well.  The  nature  of  data  collection 
work  is  sequential  tasking,  where  attention 
may  move  back  and  forth  between  different  tasks, 
but  not  focusing  on  more  than  one  at  a  time. 

♦  Training  data  collectors  requires  time  and 
effort  to  ensure  that  they  understand  their 
roles  and  responsibilities.  While  setup 
time  may  be  reduced  by  using  manual  data 
collection  methods,  personnel  requirements 
may  increase,  including  the  additional 
logistical  and  training  burden.  Most 

data  collection  efforts  are  unique  to  each 
operational  test,  meaning  that  data  collectors 
must  be  trained  for  each  test  event.  When 


employing  electronic  data  collection  devices 
such  as  PDAs,  additional  familiarization 
time  may  be  required.  Finally,  despite  being 
sufficiently  trained,  manual  data  collectors 
often  make  errors  of  omission,  transposition, 
accuracy,  or  judgment. 

When  manual  data  collection  is  the 
preferred  method,  the  test  team  should 
consider  who  is  best  suited  to  the  task. 
Using  Marines  as  data  collectors  has  some 
advantage  in  that  they  are  familiar  with 
the  military  operating  environment,  but 
data  collection  is  not  their  purpose  in  the 
Marine  Corps.  In  addition,  using  Marines 
as  data  collectors  increases  the  burden  on 
the  Operating  Forces. 

Using  civilian  data  collectors  can  lessen  the 
burden  on  the  Operating  Forces.  Civilian 
data  collectors  can  also  be  obtained  earlier 
in  the  test  planning  process  to  improve 
training  and  awareness  of  what  is  to  be 
collected  and  the  methods  for  doing  so. 
However,  a  civilian  data  collector  may 
be  inexperienced  in  the  harsh  military 
environment  and  may  be  ill-suited  for 
dealing  with  it. 

Develop  Data  Reduction  Methods 

Data  from  the  test  must  be  reduced  to  a 
form  useful  to  the  OA,  and  the  form  will 
vary  from  test  to  test.  The  formal  definition 
of  data  reduction  is  the  transformation 
of  information,  usually  empirically  or 
experimentally  derived,  into  corrected, 
ordered,  and  simplified  form.  The  term 
generally  refers  to  operations  on  either 
numerical  or  alphabetical  information 
digitally  represented,  or  to  operations  which 
yield  digital  information  from  empirical 
observations  or  instrument  readings. 

Data  reduction  methods  should  be 
documented  for  each  Measure  in  a  Test 
Plan  and  tailored  to  the  data  collection 
methods.  Following  is  a  sample  data 
reduction  method  for  preparing  to  answer 
Mean  Time  Between  Failure  (MTBF). 

The  timeline  collected  on  the  system  under 
test  must  be  reduced  to  only  the  times  that 
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apply  for  calculating  times  between  failures. 

1.  Filter  the  records  by  system  ID 

2.  Sort  all  records  in  ascending  order 
according  to  date/time 

3.  Remove  all  records  from  the  data  that 
are  not  identified  as  Operating  Time 

4.  Identify  the  arrival  date/time  of  each  TIR 
noted  as  a  system  failure  or  malfunction 

5.  Compute  the  clock  time  for  the  arrival 
of  each  failure 

6.  Compute  the  elapsed  time  between 
failures  by 

♦  subtracting  the  clock  time  for  the  start  of 
test  from  the  clock  time  of  first  failure 

♦  subtracting  the  clock  time  of  first  failure 
from  time  of  second  failure,  and  so  on 

7.  Combine  all  system  data  into  a  single 
dataset  after  the  elapsed  times  between 
failures  by  system  have  been  computed 

8.  Compute  MTBF  using  the  formula 
MTBF=  (Z  Elapsed  Times  Between 
Failures)/Z  Failures 

Test  Trials 

The  TEMP  provides  the  test  team  with 
the  basic  information  required  to  produce 
test  event  trials,  which  are  formed  around 
the  missions  Marines  will  execute  using 
the  system  under  test  and,  therefore,  may 
be  multiple  in  number.  The  test  team 
should  anticipate  that  IOT  will  cover 
every  mission  associated  with  the  system. 
Other  types  of  tests,  e.g.,  EOA  or  OA,  may 
investigate  only  one  mission  or  a  single 
capability  area  requiring  partial  execution 
of  multiple  trials. 

Adding  Detail  to  Trials 

Using  the  process  flow  brought  forward 
from  the  Test  Concept,  the  test  team 
begins  to  add  detail  to  the  trials  through 
written  instructions.  The  instructions 
include  the  actions  of  the  operators  and 
the  functions  of  the  system  as  well  as  test 


conditions  (physical,  military,  and  civil). 

The  test  team  writes  detailed  instructions 
for  a  trial  to  ensure  the  proper  placement 
and  timing  of  everything  and  everyone 
needed  for  trial  success,  relying  heavily  on 
operational  experience  and  familiarity  with 
unit  TTPs. 

Figure  3-3-7  is  an  example  based  on  a 
surveillance  system,  illustrates  the  level 
of  detail  needed  for  enemy  actions  to  be 
carried  out  as  part  of  the  test. 

Re-examine  Cause-Effect 
Relationships 

With  more  information  about  the  system 
available  since  TEMP  development,  the 
test  team  should  re-examine  the  cause- 
effect  relationship  of  factors;  the  six-Ms 
(Materiel,  Methods,  Manpower,  Machine, 
Measurement,  and  Mother-nature) 
are  more  certain  now.  For  example,  the 
required  number  of  operators  and  data 
collectors  should  be  known.  The  step-by- 
step  instructions  identify,  by  trial,  what 
testable  factors  are  to  be  systematically 
varied.  They  also  indicate  what  methods  are 
being  used  to  control  nuisance  factors  and 
what  factors  are  held  constant. 

The  test  team  writes  the  instructions 
required  to  exercise  systematic  variation  of 
the  factors  from  trial  to  trial.  The  following 
example  illustrates  the  instructions  for 
systematic  variation  of  factors  of  interest 
to  the  tester.  In  the  example,  the  tester  is 
systematically  varying  the  status  of  the 
Remote  Ground  Control  Station  (RGCS) 
monitors  and  the  types  of  role  players  in 
view  of  the  system: 

Trial  1:  Combat  Operations  Center 
(COC)-l  will  have  all  of  its  RGCS 
monitors  turned  “off,”  and  COC-2  will 
have  all  of  its  RGCS  monitors  turned 
“on.”Just  prior  to  beginning  the  event 
the  Hostile-Rifle/Scope,  Hostile-Mortar, 
Neutral,  Neutral-Rifle,  Friendly- Rifle/ 
Scope,  and  Friendly-Mortar  teams  must 
be  distributed  to  their  respective  positions. 
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Figure  3-3-8. 
Example  of  a  Trial 


Only  the  Hostile-Mortar,  Neutral- Rifle, 
and  Friendly  Mortar  teams  can  be  visible 
to  the  towers  at  the  beginning  of  trial 
#1.  The  remaining  groups  will  need  to 
be  in  position  but  hidden  in  the  terrain 
features  to  mask  their  presence  until 
they  are  needed  in  the  subsequent  trials. 
Vehicles  should  not  be  moving  during  or 
between  trials  #1, 2,  or  3.  At  the  start  of 
trial  #1,  the  vehicles  should  be  positioned 
at  points  to  affect  rapid  consolidation  and 
redistribution  of  forces  at  the  conclusion  of 
trial  #3  in  preparation  for  scenario  reset  for 
the  Dusk  Trial. 

System  Readiness 

System  readiness  is  determined  in  part 
by  reviewing  the  results  of  all  Assessment 


Reports.  These  results  (obtained  from 
Integrated  Testing  observation  and 
evaluation)  have  been  fed  back  to  the 
SEP  and  TEMP.  By  way  of  review,  the 
SEP  identifies  evaluation  questions  and 
standards  while  the  TEMP  identifies 
testing  that  should  have  occurred  before 
OT  and  OT  entrance  criteria.  By  the  time 
of  the  OTRR,  all  IOT  entrance  criteria 
must  be  satisfied. 

Resource  &  Documentation 
Readiness 

Numerous  key  resources  are  required 
to  proceed  to  an  IOT:  an  adequate  test 
team,  Operating  Forces,  test  ranges, 
training  package,  funding,  and  other 
resources.  The  OTRB  discusses  the  level 


Trial#  7 

The  trial  begins  when  the  Opposing  Force  manager  indicates  that  all  role-players  are  in  position  and  the  TM  indicates  that  both  Command 
Operations  Centers  (COC)  and  their  respective  towers  are  ready  to  commence  operations.  A  COC  and  its  towers  are  considered  ready  when  all 
cameras,  monitors,  and  communications  devices  are "on"and  pre-operations  checks  have  been  satisfied.  The  operators  will  be  given  a  threat  brief 
and  provided  with  realistic  intelligence  products  that  identify  the  threats  likely  to  be  encountered  in  their  area  of  responsibility. 

Once  the  trial  has  begun,  the  Operating  Forces  will  take  the  following  actions: 

The  Hostile-Mortar  team  will  maneuver  from  a  starting  point  (1 2STB2631 640420)  to  an  egress  point  (1 2STB2628640472)  or  hiding  point  during  the 
trial  and  remain  hidden  until  the  end  of  trial  #3. 

The  Neutral-Rifle  team  will  maneuver  in  an  area  (1 2STB2631 540421 )  and  maintain  at  least  periodic  visibility  with  the  towers  throughout 
trial  #1.  The  Neutral-Rifle  team  will  remain  in  view  of  the  cameras  during  the  first  data  collection  stop  and  then  maneuver  to  a  hiding  point 
(12STB26291 40464)  before  the  second  data  collection  stop. 

The  Friendly-Mortar  team  will  maneuver  in  an  area  (12STB2605339567)  and  maintain  at  least  periodic  visibility  with  the  towers  throughout 
trial  #1.  The  Friendly-Mortar  team  will  remain  in  view  of  the  cameras  during  the  first  data  collection  stop  and  then  maneuver  to  a  hiding  point 
(12STB26321 40414)  before  the  second  data  collection  stop. 

After  45  minutes  of  trial  time  the  Test  Manager  will  notify  the  COCs,  data  collectors,  and  0PF0R  Manager  that  the  trial  will  be  halted  for  the  first  data 
collection  stop.  During  the  5-minute  data  collection  stop  the  following  will  occur: 

The  0PF0R  Manager  will  ensure  that 

•  the  Hostile-Mortar  team  has  maneuvered  to  its  hiding  point 

•  the  Hostile-Rifle/Scope,  Neutral,  and  Friendly-Rifle  Scope  teams  emerge  from  their  hiding  points  into  the  field  of  view  of  the  cameras 

•  the  0PF0R  Controller  takes  the  appropriate  Lux  readings  and  that  this  information  is  recorded 
The  Test  Manager  will  ensure  that 

■  the  COC  &  Tower  Target  data  collectors  have  collected  their  respective  data 

•  the  COC  &  Tower  RAM  data  collectors  have  administered  the  surveys  to  the  Situation  Watch  Officers,  Watch  Officers,  and  Remote  Ground  Station/ 
Ground  Control  Station  operators. 

Upon  completion  of  the  data  collection  stop  activities  the  Test  Manager  will  signal  to  the  COCs  and  0PF0R  Manager  that  the  trial  will  now  resume 
where  it  left  off.  This  marks  the  beginning  of  trial  #2. 
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of  preparation  achieved  for  each  of  these; 

the  OTRR  verifies  that  preparations  are 
final.  SECNAVINST  5000.2  requires  the 

following  for  OTRR: 

♦  The  TEMP  is  current  and  approved.  Testing 
prior  to  Milestone  B  shall  have  an  approved 
Test  and  Evaluation  Strategy  (TES). 

♦  T&E  results  indicate  performance 
thresholds  identified  in  the  TEMP  have 
been  satisfied  or  are  projected  to  meet 
system  maturity  for  the  CDD/CPD. 

♦  All  significant  areas  of  risk  have  been 
identified  and  corrected  or  mitigation  plans 
are  in  place. 

♦  All  test  results  have  been  provided  to 
MCOTEA  by  the  OTRR,  unless  otherwise 
agreed  to  by  MCOTEA. 

♦  The  OT  entrance  criteria  in  the  TEMP 
have  been  satisfied. 

♦  System  operating,  maintenance,  and 
training  documents  have  been  provided  to 
MCOTEA  30  days  prior  to  the  OTRR 
unless  otherwise  agreed  to  by  MCOTEA. 

♦  Logistical  support  is  available  as 
documented,  including  spares,  repair  parts, 
and  ground  support  equipment. 

♦  Operating  Forces  manning  the  system 
are  adequate  in  number,  rank,  MOS,  and 
experience  to  simulate  normal  operating 
conditions. 

♦  Training  has  been  completed  and  is 
representative  of  that  planned  for  fleet  units. 
Note:  The  Marine  Corps  routinely  waives 
this  requirement  so  that  New  Equipment 
Training  is  conducted  just  before  the  Pilot 
Test;  however,  the  Training  Plan  should  be 
in  place  by  the  OTRR. 

♦  All  resources  and  funding  required  to 
execute  OT  are  identified  and  available, 
including  instrumentation,  simulators, 
targets,  and  expendables. 

♦  Models,  simulators,  and  targets  are 
accredited  for  intended  use.  Note: 
MCOTEA  requires  M&S  accreditation 
to  be  completed  by  the  OTRB.  The  system 
provided  for  OT&E,  including  software,  is 


production-representative. 

♦  Differences  between  the  system  provided 
for  test  and  production  configuration  are 
addressed. 

♦  Threat  information  is  available  (e.g.,  threat 
system  characteristics  and  performance, 
electronic  countermeasures,  force  levels, 
scenarios,  and  tactics),  to  include  security 
classification. 

♦  System  is  safe  to  use  as  planned  in  the 
Concept  of  Employment  and  the  PM  has 
provided  the  appropriate  safety  releases.  Any 
restrictions  to  safe  employment  are  stated. 

♦  Environmental,  Safety,  and  Occupational 
Health  (ESOH)  program  requirements  are 
satisfied.  The  system  complies  with  Navy/ 
Marine  Corps  ESOH/hazardous  waste 
requirements,  where  applicable.  ESOH/ 
hazardous  waste  reviews  and  reports  have 
been  provided  to  Director,  MCOTEA. 
When  an  energetic  is  employed  in  the 
system,  Weapon  System  Explosive  Safety 
Review  Board  criteria  for  conduct  of  test 
have  been  met. 

♦  All  software  is  sufficiently  mature  and  stable 
for  introduction  into  the  Marine  Operating 
Forces.  All  software  Trouble  Reports  are 
documented  with  appropriate  impact  analyses. 
There  are  no  outstanding  Trouble  Reports  that 

♦  Prevent  the  accomplishment  of  an 
essential  capability 

♦  Jeopardize  safety,  security,  or  other 
requirements  designated  “critical” 

♦  Adversely  affect  the  accomplishment 
of  an  essential  capability  and  no 
workaround  solution  is  known 

♦  Adversely  affect  technical,  cost,  or 
schedule  risks  to  the  project  or  to  life- 
cycle  support  of  the  system,  and  no 
workaround  solution  is  known 

♦  For  software  qualification  testing,  a 
Statement  of  Functionality  that  describes 
the  software  capability  has  been  provided  to 
Director,  MCOTEA. 

♦  For  programs  with  interoperability 
requirements  (e.g.,  information  exchange 
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requirements  in  ICD/CDD/CPDs), 
the  appropriate  authority  has  approved 
the  ISP  and  JITC  concurs  that  program 
interoperability  demonstrated  in 
development  has  progressed  sufficiently  for 
the  phase  of  OT  to  be  conducted. 

♦  For  spectrum  management,  the  Stage  3 
“Developmental”  DD-1494  (at  a  minimum) 
is  in  place. 

♦  For  IT  systems,  including  NSS,  the  system 
has  been  assigned  a  Mission  Assurance 
Category  (MAC)  and  Confidentiality 
Level.  System  certification  accreditation 
documents,  including  the  Phase  2  System 
Security  Authorization  Agreement  and  the 
Interim  Authority  to  Test  (IAT),  Interim 
Authority  to  Operate  (IATO),  or  platform 
IT  designation  letter,  as  applicable,  have 
been  provided  to  MCOTEA. 


Verification,  Validation,  and 
Accreditation  of  Models  and 
Simulations 

MCOTEA  must  accredit  any  model  or 
simulation  used  to  supplement  data  in  a 
MCOTEA  assessment  or  test  or  used  in 
any  MCOTEA  analysis.  See  chapter  6  for 
an  in-depth  discussion  of  M&S  selection 
and  accreditation. 

Operational  Risk  Management 

The  OTPO  is  directly  responsible  for 
the  safe  conduct  of  all  operational  test 
events.  During  planning,  safety  must  be 
addressed  in  the  following  areas:  Safe  and 
Ready  Certification  for  the  system  under 
test;  training  to  ensure  that  the  system 
is  operated  safely;  and  that  operations 
occur  in  accordance  with  local  range  and 
base  procedures.  These  issues  require  an 
Operational  Risk  Management  (ORM) 
assessment  for  each  event  the  test 


Note:  References  for  this  section  appear  at  the 
end  of  the  chapter. 
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Annex  A:  Test  Logistics  Support 


Requirements  for  Other- 
Service  Assets 

Current  operational  tempo  determines 
availability  of  Other-Service  assets,  so  they 
may  be  difficult  to  procure. 

Navy  Assets 

If  a  test  requires  Navy  assets,  particularly 
amphibious  ships  or  landing  craft,  the  test 
team  must  obtain  a  Test  and  Evaluation 
Identification  Number  (TEIN)  as 
described  in  SECNAVINST  5000.2.  A 
specific  format  exists,  and  it  must  be  sent 
via  the  appropriate  Requirements  office 
(DC,  CD&I)  for  endorsement.  With  the 
TEIN  in  hand,  the  test  team  completes 
the  Fleet  Support  Request  (FSR)  form. 
Since  the  East  Coast  and  West  Coast 
Fleet  Commands  differ  in  scheduling 
lead  times,  the  test  team  must  contact  the 
appropriate  scheduling  coordinators  at  least 
6  months  in  advance  to  be  included  on  the 
scheduling  conference  notification  lists. 
(This  can  be  accomplished  by  contacting 
the  current  OPNAV  N912C  Project 
Officer,  who  answers  to  the  OPNAV  N091 
scheduler.)  The  FSR  usually  needs  to  be 
submitted  6  months  ahead,  but  the  actual 
scheduling  conference  may  occur  within 
3  months  to  1  month  of  the  test  date.  If 
another  Service  is  the  lead  OTA,  and  the 
Marine  Corps  is  the  only  party  with  an 
amphibious  mission,  the  test  team  may 
have  to  schedule  amphibious  operations 
testing  independent  of  the  lead  OTA. 

Army  Air  Assets 

Air  asset  requirements  present  unique 
challenges.  If  a  test  needs  Army  Air,  such 
as  a  CH-47,  to  demonstrate  internal  or 
external  lift  capabilities,  the  test  team 
should  consider  the  Army  Reserves  or 
the  Air  National  Guard.  The  test  team’s 
POC  with  the  Army’s  OTA  may  be  able 


to  provide  contact  names  and  telephone 
numbers.  If  MCOTEA  is  the  lead  OTA 
and  Army  assets  are  required,  they  must 
be  requested  through  the  Operational 
Test  Command  via  the  Outline  Test  Plan 
(OTP),  drafted  by  the  Army  POC.  (Note: 
Modifying  the  OTP,  once  published,  is  a 
difficult  and  slow  process.) 

Marine  Air  Assets 

The  test  team  can  usually  coordinate  the 
use  of  both  fixed-wing  and  rotary- wing 
aircraft  through  the  Marine  Aircraft 
Liaison  Officer  (ALO)  for  the  respective 
MEF.  The  MEF  normally  assigns  the 
duties  to  a  specific  squadron  and  issues 
the  DIRLAUTH  for  detailed  planning. 
MV-22  Osprey  support,  however,  proceeds 
differently.  If  Wing  assets  are  stood  up 
and  available,  planning  will  proceed 
through  the  ALO  at  MEF.  If  a  test  will 
use  planes  from  VMX-22,  the  test  team 
should  coordinate  with  the  squadron  itself, 
since  they  work  directly  for  DC  AIR, 
not  the  MEF.  The  MEF  G-3  will  also  be 
required  to  issue  an  authorization  for  FMF 
Marines  to  fly  in  VMX-22  aircraft,  since 
the  Marines  belong  to  MEF  and  not  the 
aircrafts’  command.  Scheduling  the  Wing 
may  require  flexibility,  so  the  test  team 
should  provide  alternate  dates/times  in 
the  Test  Plan.  (Note:  The  test  team  should 
also  consider  that  the  qualifications  and 
certifications  of  both  the  pilots  and  the 
ship  affect  whether  the  schedule  requires 
shipboard  landings.  The  pilots  cannot 
take  off  from/land  on  the  ship  if  their 
qualifications  are  not  current.) 

Air  Force  Assets 

Air  Force  lift  assets  can  often  be  arranged 
through  the  ALO  at  the  MEF  level. 

The  Marine  ALO  can  provide  contact 
names  and  numbers  and  may  be  willing  to 
perform  the  necessary  coordination. 
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Equipment  Requirements 
and  Test  Site  Coordination 

Although  a  published  FOS  has  identified 
the  Marine  Corps  forces  (MARFOR) 
(LANT  or  PAC)  that  will  support  the 
OT,  it  has  not  necessarily  defined  the  test 
location.  Depending  on  the  nature  of  the 
test,  Camps  Lejeune  and  Pendleton  may 
suit  a  portion  of  the  data  requirements,  but 
they  seldom  provide  the  extremes  needed. 
Twentynine  Palms  is  popular  for  desert/ 
hot  weather  testing,  but  alternate  locations 
may  be  less  crowded.  Several  Army  Reserve 
and  National  Guard  sites  offer  adequate 
facilities  for  temperate  and  cold  weather 
testing,  such  as  Camp  Ripley  in  Minnesota 
and  Fort  Pickett  in  Virginia.  Time  of 
year  is  a  factor:  Reserve  and  Guard  units 
book  their  facilities  from  May  through 
September  for  their  2-week  training 
evolutions.  Other  government  and  civilian 
agencies  are  also  potential  candidates. 
Nevada  Automotive  Test  Center  in  Nevada 
is  an  excellent  motor  vehicle  test  site. 

Other  options  include  Yuma  Proving 
Ground,  Aberdeen  Test  Center,  and  Naval 
Surface  Warfare  Center,  Dahlgren.  The  S-3 
has  access  to  the  capabilities  of  various  test 
and  training  ranges  around  the  country 
and  should  be  consulted  before  deciding  on 
the  optimal  test  venue  for  each  test.  When 
using  government  labs,  the  test  team  must 
obtain  a  Rough  Order  Magnitude  (ROM) 
cost  proposal  because  these  labs  can  be 
expensive.  For  all  test  sites,  the  test  team 
must  generate  a  ROM  for  site  cleanup 
after  the  test. 

Although  the  test  team  may  begin 
informal  identification  and  coordination 
with  the  test  site,  formal  coordination  is 
accomplished  by  the  S-3.  The  S-3  will 
notify  the  test  team  of  the  test  site  POC 
after  formal  coordination  is  complete. 

At  that  time,  the  test  team  will  assume 
responsibility  for  coordinating  with  the  test  site. 

After  selecting  the  test  sites,  the  test 
team  should  communicate  with  a 


POC  via  telephone/e-mail  to  ascertain 
documentation  requirements  and  to 
schedule  a  site  visit.  Although  some  details 
can  be  resolved  over  the  phone,  face-to- 
face  contact  ensures  clear  communication. 
Traveling  with  a  representative  from  the 
PM  is  advantageous  because  scheduling 
training  facilities  and  assets  at  once  can 
save  time  and  money.  MOIC  attendance 
on  these  visits  is  strongly  encouraged. 
During  the  site  visit,  the  test  team 
should  attempt  to  establish  POCs  for 
billeting,  messing,  ranges/training  areas, 
ammunition  support  (if  needed),  and 
network  connectivity  and  should  identify 
any  special  waivers,  certifications,  or 
area-peculiar  requirements  (e.g.,  OIC/ 
RSO)  certifications,  port-a-johns  in 
the  field,  dunnage  collection  schedules/ 
costs,  frequencies  and  radios,  waivers  for 
privately  owned  vehicles  in  the  training 
areas,  etc.  If  the  program  involves  classified 
documentation  or  equipment,  advance 
coordination  for  delivery  and  storage  is 
mandatory.  If  the  test  team  coordinates 
ammunition  delivery  procedures  in 
advance,  the  process  will  be  simplified  as 
the  test  dates  draw  closer.  The  test  team 
should  plan  to  visit  the  test  site  at  least 
once  more  after  the  initial  visit  and  before 
the  test  to  finalize  and  confirm  details 
previously  arranged. 

Identification  of 
Required  Facilities  and 
Logistics  Support 

The  S-4  helps  the  test  team  coordinate 
on-site  logistics  support  for  MCOTEA 
tests.  Site  visits  enable  the  test  team  to 
identify  and  consolidate  administrative 
and  logistics  support.  Office  space  and 
equipment  are  most  commonly  needed. 
Sometimes  one  source  can  address  phone, 
fax,  and  copier  requirements  as  well,  but 
the  test  team  may  benefit  from  shipping 
the  items  from  MCOTEA  to  the  test  site. 
Maintenance  spaces  are  another  frequent 
issue,  and  if  weapons,  classified  documents/ 
items,  or  serialized  equipment  are  involved, 
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armory  or  other  secure  storage  facilities 
are  key  requirements.  If  training  will  be 
conducted  immediately  preceding  the  OT, 
the  PM  representative  will  be  interested 
in  scheduling  classroom  spaces,  and  the 
test  team  will  need  a  place  to  conduct  data 
collector  training. 

Data  Collection  Forms 

One  of  the  Data  Manager’s  largest 
responsibilities  is  creating  the  Data 
Collection  Forms  that  will  be  used  during 
EOA/OA/OT  execution  and  the  final 
evaluation.  The  forms  may  be  electronic 
(created  and  used  in  a  portable  data 
collection  device)  or  paper-based  and 
filled  in  manually.  If  data  is  collected  and 
stored  only  in  electornic  form,  a  backup 
of  the  data  must  be  created  as  soon  as 
practical  to  protect  against  data  loss  due  to 
electronic  malfunction.  Working  from  the 
Evaluation  Framework,  the  Data  Manager 
develops  forms  to  collect  each  program’s 
data  requirements  and  to  resolve  all  defined 
Measures. 

Types  of  Forms 

The  DM  creates  forms  to  capture  all  of 
the  requirements  outlined  in  the  Test  Plan. 
Form  structure  is  based  on  the  types  of 
Measures  contained  in  the  SEP.  The  relation 
of  Measures  to  forms  is  illustrated  below. 

Quantitative  forms  collect  numerical  data, 
e.g.,  RAM  and  TIRs. 

Qualitative  forms  collect  the  ratings  and 
comments  of  the  operators  and  SMEs  and 
are  written  as  Survey  Questions  (see  right). 

Verification  Forms  collect  data  for 
the  purpose  of  proving  that  items  exist 
or  are  included  with  the  system  to  be 
tested.  Additional  forms  may  be  created 
to  characterize  the  operational  test 
environment.  While  each  form  may  be 
adapted  to  the  particular  event,  certain 
reference  information  must  appear  on 
every  form:  e.g.,  the  item  being  tested, 
operator  ID,  date,  and  time.  From  there 
the  forms  are  designed  to  capture  requisite 


information:  for  example,  Test  Incidents, 
RAM,  Maintenance,  Demographics,  and 
Operations  Log.  Other  forms  that  could 
be  developed  include  Inventory  Control, 
Weather  Log,  Information  Assurance,  and 
Crew  Assessment.  While  a  few  basic  forms 
(Operations  Log,TIR,  and  Weather)  may 
be  similar,  most  of  the  forms  must  be  built 
to  capture  program-specific  data  to  answer 
the  Measures  in  the  SEP. 

The  DM  must  ensure  that  the  forms  flow 
logically  and  are  easy  for  a  data  collector 
in  the  field  to  follow.  Each  set  of  forms  is 
program-specific  and  will  vary  greatly  in 
design  and  depth  of  data  collected. 

Survey  Questions 

Survey  questions  are  the  primary  method 
of  collecting  qualitative  data;  each 
qualitative  Measure  has  questions  assigned 
to  it.  The  DM  works  with  the  test  team 
and  an  SME  (e.g.,  the  Human  Factors  and 
Safety  SME)  to  develop  the  questions. 

The  basis  for  questions  can  derive  from  the 
SEP,  the  Request  for  Clarifications,  and  the 
OMS/MPs.  Another  option  for  creating  a 
survey  is  to  perform  a  structured  interview, 
in  effect  an  open  forum  that  asks  the 
operators  to  state  their  opinions  about  the 
system  in  a  structured  way. 

MCOTEA  prefers  using  more  quantitative 
data  sources,  but  surveys  can  be  useful  in 
finding  issues  for  further  analysis  and  in 
helping  to  identify  risks. 

Data  Collector  Training 

Data  Collector  (DC)  training  is  the 
opportunity  to  provide  instruction  to  the 
collection  team  on  the  purpose  of  the  test 
and  their  role  in  it.  DC  training  is  usually 
done  at  the  test  site  after  the  arrival  of  all 
personnel.  This  should  occur  a  couple  of 
days  before  the  Pilot  Test. 

Everyone  should  understand  that  the 
purpose  of  the  test  is  NOT  to  make  the 
system  work,  but  to  obtain  unbiased  data 
on  its  performance,  given  the  crew  training 
and  operating  conditions  particular  to  the 
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event.  DCs  should  understand  that  they  are 
to  gather  the  data  requested  on  the  forms, 
but  not  attempt  to  analyze  or  interpret  the 
data  or  interfere  with  operators  using  the  system. 

Data  collector  training  focuses  on  training 
the  DCs  to  accomplish  their  mission  in 
the  test.  This  includes  going  over  each 
data  form  in  detail,  paper  or  electronic. 

The  instructors  are  usually  the  test 
team  members  responsible  for  creating 
the  forms.  A  substantial  portion  of  the 
training  should  be  dedicated  to  practical 
application.  If  automated  data  collection  is 
employed,  the  instrumentation  supporting 
the  automation  should  be  used  as  an 
integral  part  of  this  training.  The  team 
should  discuss  the  forms  with  the  DCs 
and  solicit  their  recommendations  on 
such  items  as  terminology,  so  that  changes 
can  be  made  and  validated  with  the  DCs 
before  the  test  begins.  The  instructors 
should  make  notes  of  all  questions  asked 
and  the  responses  given  by  the  instructors 
to  aid  in  consistency  throughout  the  test. 
The  Data  Collector  Handbook  should  be 
covered  in  the  training.  DCs  may  then  use 
this  reference  book  throughout  the  test. 

Environmental 
Considerations  for  Data 
Collection 

Data  collection  efforts  on  an  operational 
test  must  occur  in  day  or  night,  rain  or 
shine,  wet  or  dry,  cold  or  hot,  etc.  The 
operating  environment  will  impact  the 
choices  of  data  collection  methods.  Things 
to  consider  when  choosing  data  collection 
methods  include: 

♦  Visibility  (natural  light  or  availability  of 
sources  of  artificial  light).  Data  collection 
under  low  light  or  no  light  situations 
presents  unique  challenges.  Depending 
on  the  method  of  collection,  paper  for 
example,  a  data  collector  would  need  an 
artificial  source  of  light  to  collect  data.  Care 
should  be  taken  to  ensure  that  the  artificial 
light  source  does  not  interfere  with  the 
operations  of  the  system  under  test  or  its 
operators.  When  using  electronic  means  to 


collect  data,  the  same  holds  true,  except  that 
the  electronic  means  are  often  sources  of  light. 

♦  Precipitation  (rain,  freezing  rain,  sleet,  snow, 
none).  Depending  on  the  environment, 
data  collection  methods  need  to  be  resistant 
to  precipitation.  Waterproof  paper  and 
ruggedized  data  collection  devices  are 
available  to  protect  data  collection  efforts. 

♦  Temperature  (cold/hot).  Cold  and  hot 
environments  can  make  data  collection 
difficult.  Electronic  devices  can  fail  in 
extreme  cold  and  heat.  Likewise,  clothing 
designed  for  inclement  weather  may  make 
paper  data  collection  difficult  to  accomplish. 

♦  Data  Collection  Mobility.  Another  serious 
consideration  is  whether  data  must  be 
collected  on-the-move.  Movement  by  foot 
or  vehicle  can  make  collecting  data  very 
difficult.  It  is  difficult  to  write  or  tap  touch 
screens  effectively  while  on-the-move. 

Data  Collection  Based  on 
Data  Requirements 

What  is  being  measured  and  the  data 
requirements  themselves  often  dictate  how 
the  data  is  to  be  collected.  For  example,  if 
“elapsed  time”  was  the  data  requirement, 
then  the  analyst  may  choose  to  instrument 
the  trial  with  a  stopwatch.  However,  if 
“Time  Start”  and  “Time  Stop”  are  the  data 
requirements,  then  the  analyst  may  choose 
to  instrument  the  trial  with  a  device  that 
creates  time  stamps  for  events,  such  as  a 
ruggedized  PDA. 

Building  a  Data  Repository 

Once  all  data  requirements  have  been 
developed,  the  DM  builds  an  electronic 
data  repository,  an  electronic  medium 
for  storing  the  collected  data.  The 
preferred  method  is  a  database,  although 
spreadsheets  may  be  used  for  smaller  tests. 
All  test  data,  including  the  data  collected 
on  paper  forms,  must  be  placed  into  the 
data  repository  for  appropriate  analyses. 
The  repository  must  be  able  to  support  the 
analytic  requirements  of  every  Measure  for 
the  test;  if  data  to  support  every  Measure 
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is  not  included  in  the  repository,  the 
repository  is  inadequate. 

Maintaining  Data 
Integrity  and  Security 

The  test  team  DM  is  responsible  for 
maintaining  data  integrity  (completeness, 
correctness,  and  noting  caveats  associated 
with  data  elements)  and  security  (no 
unauthorized  changes).  Limiting  access  to 
the  repository  through  password  protection 
maintains  data  security  as  does  limiting 
write  privileges  inside  the  repository. 

Data  Collection  Verification 
and  Validation 

The  test  team  verifies  the  adequacy  of 
the  data  collection  plan  designed  for 
the  system  under  test  and  validates 
the  accuracy  and  completeness  of  the 
resulting  data  in  reporting  the  Test  Plan 
Measures.  Data  collection,  including  the 
collection  equipment,  should  be  verified 
and  validated  before  use  in  the  actual  test. 
Data  Collection  (DC)  V&V  is  performed 
once  data  collection  methods  and  the 
data  repository  have  been  constructed. 
Accordingly,  the  test  team  plans  and 
conducts  a  DC  V&V  exercise  (not  to  be 
confused  with  a  VV&A)  that  tests  the  data 
collection  methodology  and  ensures  that 
the  data  collection  equipment  functions 
properly  and  reliably. 

All  systems  that  will  support  data 
collection  for  the  operational  test  must  be 
programmed  and  present  at  the  DC  V&V 
(automated  data  collection  devices,  survey 
computers,  primary  forms,  etc.).  The  DC 
V&V  should  include  as  many  members 
of  the  test  team  as  permissible,  but  at  a 
minimum  the  individual  responsible  for 
data  collection  during  the  test,  the  TM,  and 
either  the  MCOTEA  DM  or  OA  should 
be  present.  Following  the  DC  V&V,  if  the 
test  team  does  not  discover  any  issues,  the 
items  should  be  ready  to  ship  to  test.  If 
the  test  team  discovers  issues,  they  should 
repeat  the  DC  V&V  following  corrections 


(the  test  team  can  tailor  the  DC  V&V  to 
focus  on  the  issues  they  discover).  The  DC 
V&V  consists  of  the  following  four  phases: 

Phase  I 

Cross  reference  the  data  requirements  with 
the  data  collection  media  to  ensure  all  data 
requirements  are  addressed.  Distribute  a 
draft  Data  Collector  Handbook  to  test 
team/V&V  participants  and  conduct  data 
collection  training. 

Phase  II 

Distribute  a  DC  V&V  plan  to  simulate  test 
events.  The  DC  V&V  plan  shall  require 
that  at  least  one  participant  touches  every 
possible  button  in  the  electronic  forms 
as  well  as  enter/exit  forms  from  every 
potential  entry/exit  point.  Using  the  DC 
V&V  plan,  the  participants  will  enter  the 
simulated  test  data  into  the  automated  data 
collection  devices  for  each  Measure. 

Phase  III 

Distribute  a  set  of  scripted  answers  to 
survey  questions  and  have  the  participants 
log  in  to  the  survey  database  and  enter 
the  scripted  responses.  Ensure  that  every 
survey  session  and  respondent  billet  is 
accessed.  For  any  sessions  that  will  have 
multiple  respondents  during  testing,  ensure 
that  multiple  participants  take  the  survey 
during  the  DC  V&V.  Also  ensure  that  a 
mix  of  instances  exist  where  respondents 
select  the  same  response  and  different 
responses  in  the  rating  scale. 

Phase  IV 

Download  all  data  into  the  data  repository 
and  invalidate  any  inappropriate  records. 

Run  all  reports  and  review  to  ensure  that 
they  work  properly  (at  a  minimum,  a  report 
should  exist  for  every  test  MOE,  MOP,  and 
MOS).  When  RAM  are  included  in  the  test 
MOSs,  additional  reports  are  required.  These 
reports  include  system  timelines,  TIRs,  and 
maintenance  data.  Additionally,  the  process 
of  scoring  and  reporting  test  incident  report 
should  be  V&Vd  as  part  of  the  process. 
Upon  completion  of  the  DC  V&V,  the  test 
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team  reports  to  the  MCOTEA  Division 
Head  the  results  and  if  necessary  any  steps 
to  correct  problems. 

Test  Site  Visit 

Even  if  the  test  team  has  visited  the  test 
site  earlier  in  the  planning,  another  visit 
should  occur  at  least  2  weeks  before  test  to 
confirm  the  following: 

♦  dining  and  sanitary  facilities  are  ready 

♦  range  regulations  have  not  changed 

♦  corpsman  is  available,  if  needed 

♦  all  shipping/receiving  details  are  arranged 

♦  coordination  with  key  staff  officers  in  the 
host  organizations  and  the  Base  Public 
Affairs  Office  has  occurred 

♦  other  range  users  and  stakeholders  know 
how  the  test  may  affect  them  (range 
closures,  etc.) 

Check  Equipment/ 
Instrumentation  Operation 

To  prevent  delays  once  testing  begins,  the 
test  team  should  arrange  to  have  limited 
technical  inspections  (LTI)  and  operations 
checks  for  all  major  test  support  systems 
and  equipment  before  the  items  are 
transported  from  the  providing  commands 
to  the  test  site.  This  can  be  as  simple  as 
ensuring  that  a  generator  is  working  or  a 
road  wheel  on  a  vehicle  will  last  for  the 
duration  of  the  test.  No  equipment  should 
arrive  at  the  test  site  that  may  require 
major  preventive  maintenance  during 
test.  Specific  equipment  configuration 
requirements  should  also  be  confirmed. 

Instrumentation 

Rehearsals  of  instrumentation  setup, 
operation,  and  teardown  should  be 
conducted  at  least  2  weeks  before  test. 
Validation  and  data  reduction  procedures 
for  video  data  should  be  rehearsed  before 
the  Pilot  Test,  allowing  adequate  time  to 
adjust  instrumentation  schematics  and 
collection  plans,  if  necessary. 


Transporting  the  Test  Team 
and  Test  Equipment 

The  S-4  helps  the  test  team  coordinate 
transportation  to  the  test  site.  If  many 
test  participants  are  involved  and  the  site 
is  not  within  motor  transport  range,  air 
transportation  becomes  the  most  viable 
option.  The  ALO  at  MEF  can  assist  here. 
Although  C-130  transport  (USMC  or  Air 
Force)  is  ideal,  these  aircraft  are  usually 
overbooked  and  unavailable.  Air  Force 
transport  (C-5,  C-17,  etc.)  is  possible: 
the  ALO  may  be  able  to  coordinate  with 
the  Air  Force  counterpart  to  inquire  into 
aircraft  availability.  Commercial  charter 
transportation  may  be  the  best  option. 

The  test  team  should  coordinate  with  the 
Traffic  Management  Office  (TMO)  and 
provide  a  detailed  roster,  but  this  requires 
travel  orders  per  Fiscal’s  guidance.  Local 
transportation  at  the  embarkation  and 
debarkation  points  must  still  be  arranged, 
but  the  local  base  transportation  office  can 
provide  buses  (military  or  civilian)  for  that 
purpose.  In-and-around  transportation  will 
depend  on  the  size  of  the  test  contingent. 
For  groups  of  less  than  50,  test  participants 
can  drive  rental  vans.  A  regular  bus 
schedule  can  be  arranged  through  the 
Regional  Transportation  Facility  (RTF)  for 
larger  contingents. 

Note:  If  the  test  team  uses  commercial 
(rental)  vans,  the  OTPO  must  procure 
a  release  from  the  RTF  stating  that 
government  vans  are  not  available.  Upon 
receipt,  the  local  Base  Comptroller  will 
generate  a  contract  so  that  the  Marine  test 
participants  will  not  be  charged. 

Travel  Orders 

Travel  Orders  for  test  participants, 
should  they  be  needed,  can  be  handled 
in  two  ways:  MCOTEA  (Fiscal)  can 
cut  the  orders  for  each  individual  or  the 
appropriate  data  can  be  sent  to  the  test 
unit’s  administrative  section  and  the  orders 
prepared  there. 
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Transporting  Test  Equipment 

Normally  the  PM  is  responsible  for 
transporting  the  test  equipment  to  and 
from  the  test  sites.  The  test  team  should 
coordinate  with  the  PM’s  representative  to 
arrange  for  the  equipment’s  timely  arrival 
on  location.  After  equipment  arrival  at 
the  test  site,  the  test  team  should  conduct 
a  joint  LTI  (with  PM  and  MCOTEA 
representatives)  to  ensure  that  nothing 
was  damaged  in  transit.  If  training  is 
scheduled  for  immediately  before  OT,  the 
PM  will  probably  need  to  use  maintenance 
facilities  to  prep  the  articles.  MCOTEA 
test  equipment  destined  for  the  test  site, 
including  any  electronic  data  collection 
devices  and  laptops,  is  usually  boxed 
in  secure  shipping  containers  and  sent 
via  TMO.  The  test  team  can  obtain  the 
requisite  documentation  from  the  S-l, 
including  documentation  for  the  return  trip. 

Site-Specific  Restrictions 

When  arranging  travel  plans,  the  test  team 
must  consider  site-specific  restrictions.  For 
example,  winter  travel  to  the  Cold  Regions 
Test  Center  (CRTC)  in  Alaska  includes  a 
flight  to  Fairbanks  and  a  drive  to  CRTC. 
However,  travelers  must  remain  overnight 
in  Fairbanks  if  they  arrive  after  1500 
because  authorities  discourage  traveling 
the  100  miles  in  the  icy  darkness.  The  test 
team  must  locate  adequate  billeting  for 
any  test  participants  arriving  after  1500.  In 
addition,  special  permission  is  required  for 
Marine  use  of  15-passenger  vans. 

Marine  Corps  Equipment 

Finally,  the  test  team  must  consider  the 
availability  of  routine  Marine  Corps 
equipment.  If  a  host  unit  is  assigned  as 
test  support,  that  unit  normally  provides 
required  assets,  i.e.,  MTVRs,  HMMWVs, 
weapons  (M2,  MK19,  etc.),  radios,  etc. 
MCOTEA  covers  repairs,  fuel,  etc.,  as 
test  costs.  If  no  host  unit  exists,  the  test 
team  should  inquire  into  the  existence  of 
an  equipment  allowance  pool,  such  as  the 


one  atTwentynine  Palms.  A  good  FTI  will 
help  keep  repair  costs  lower  at  the  end  of 
the  test.  The  FOS  should  have  identified 
these  assets,  and  discussions  with  higher 
headquarters  during  the  planning  process 
should  have  identified  the  source. 

Test  Funding 

During  the  site  coordination  visit,  the 
test  team  must  visit  the  base/facility 
comptroller  to  identify  a  POC.  At  most 
bases  the  test  funds  will  be  MIPR’d 
(Military  Interdepartmental  Purchase 
Request)  to  the  comptroller,  who  will  be 
the  central  paymaster  for  test  expenses. 
However,  the  Base  comptroller  cannot 
cross  accounts,  meaning  that  Base  can 
cover  expenses  that  most  functional 
areas  generate  except  those  related  to  the 
Marine  Division.  If,  for  example,  host 
unit  equipment  (a  Division  asset)  needs 
repair,  those  funds  must  be  filtered  through 
the  Division  Comptroller.  MCOTEA 
needs  clarification  of  the  various  expense 
channels  as  early  as  possible,  so  the  test 
team  must  provide  the  contact  information 
(POC,  telephone,  and  fax  numbers)  to  the 
MCOTEA  Fiscal  section. 
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Step  4:  Operational  Test  Execution 


New  Equipment  Training 

New  Equipment  Training  (NET), 
including  maintenance  training,  is  typically 
the  first  official  event  of  the  OT  and  should 
occur  immediately  before  the  Pilot  Test. 

It  is  the  only  OT  event  that  involves  the 
PM  and  is,  in  fact,  the  PM’s  responsibility. 
Although  directed  to  the  operators  and 
maintainers  participating  in  the  test,  all 
test  team  members  should  attend  NET. 
Any  materials  used  in  the  NET  and 
subsequent  operational  test  must  conform 
to  the  requirements  of  MCO  P5215.17. 
Operational  Test  must  not  begin  until 
operators  and  maintainers  are  properly 
trained  on  the  functions  of  the  system  and 
can  use  it  in  an  operational  environment. 

Test  Execution 

Pilot  Test 

After  NET  and  immediately  before  the 
Record  Test,  the  test  team  executes  the 
Pilot  Test,  which  functions  as  both  a 
rehearsal  and  a  readiness  check  for  the 
Record  Test.  The  Pilot  Test  events  should 
mirror  those  of  the  Record  Test.  While 
not  required,  it  makes  sense  to  use  scenario 
elements  from  the  Record  Test  to  build  the 
Pilot  Test  plan.  Daytime  and  nighttime 
events  should  be  executed  over  the  same 
or  similar  terrain  as  that  of  the  Record 
Test.  Data  should  be  collected  using  the 
same  electronic  data  collection  devices 
or  paper  forms  that  will  be  used  during 
the  Record  Test.  Rarely  occurring  events 
such  as  unscheduled  maintenance  should 
be  scripted  into  the  Pilot  Test  scenario. 
Inserting  special  events  into  the  Pilot  Test 
validates  test  elements  such  as  escalating 
maintenance  procedures  associated  with 
unscheduled  maintenance  events  and  the 
TIR  collection  process. 

During  the  Pilot  Test,  the  OTPO/ 

TM  should  assess  test  unit  and  data 
collection  unit  performance  to  confirm 


the  adequacy  of  the  NET.  If  electronic 
data  collection  devices  are  used,  data 
should  be  downloaded  to  the  appropriate 
database  and  reports  run  to  ensure  that  the 
process  is  capturing  the  required  data  and 
that  data  collectors  are  properly  entering 
information.  The  Pilot  Test  phase  ends 
only  when  the  test  team  personnel  and 
MOIC  are  confident  that  the  test  unit 
and  data  collection  team  are  fully  prepared 
to  execute  their  responsibilities  and  all 
support  elements  for  test  execution  are  in 
place.  If  the  OTPO/TM  and  MOIC  are 
not  confident  about  any  element  of  test 
execution  (system  operators,  test  personnel, 
data  collection  process,  or  equipment),  they 
must  take  corrective  action  and  conduct 
another  Pilot  Test.  Record  Test  must  not 
begin  until  all  elements  of  test  execution 
are  satisfactory. 

Record  Test 

The  Record  Test  is  the  culmination  of 
all  test  planning  activities;  it  executes 
the  Test  Plan  and  accurately  collects 
the  resulting  test  data.  The  Record  Test 
generates  daily  data  results  and  Situation 
Reports  (SITREP).  Data  results  eventually 
populate  the  Test  Data  Report.  The 
SITREPs  must  contain,  at  a  minimum, 
the  planned  trials  and  those  that  were 
performed,  the  planned  data  collection  and 
the  data  actually  collected,  and  problems 
with  executing  the  test. 

Test  Plan  execution  is  a  team  effort.  Test 
team  personnel  and  the  MOIC  of  the 
Operating  Forces  must  continuously 
coordinate  their  activities  to  ensure  that 
all  test  events  are  executed  and  that  all 
necessary  data  is  collected.  If  necessary,  the 
test  team  must  take  the  time  to  adjust  the 
schedule  to  ensure  that  all  test  and  data 
collection  objectives  are  met  during  the 
Record  Test,  not  afterwards.  This  constant 
coordination  often  results  in  long  days 
for  the  test  team,  who  will  arrive  first  and 
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depart  last  each  day.  At  each  day’s  end, 
the  test  team  reviews  that  day’s  events  and 
data  collected,  completes  and  forwards 
the  SITREPs  to  the  Branch  and  Division 
Head  as  well  as  others  as  directed,  and 
prepares  for  the  next  day’s  schedule.  The 
OAs  may  spend  much  of  each  night 
reviewing  the  data  collected  during  the 
day’s  testing  and  report  the  status  of  data 
collection  the  next  morning  at  the  daily 
brief.  The  review  may  be  done  both  on¬ 
site  and  at  remote  locations,  including 
MCOTEA  headquarters.  Test  teams 
should  plan  for  and  expect  to  send  data 
(electronically)  to  MCOTEA  headquarters 
once  per  day  during  active  testing.  A 
successful  Record  Test  results  from  good 
planning,  flexible  execution,  continuous 
coordination,  and  hard  work. 

Data  Reduction 

Data  reduction,  while  technically  a  posttest 
activity,  in  reality  begins  during  Pilot  Test 
and  continues  throughout  Record  Test. 
Initial  analysis  may  be  performed  as  data 
is  reduced,  but  these  results  are  of  limited 
value  because  each  subsequent  data  point 
obtained  has  the  potential  to  change  the 
analytic  results.  Therefore,  the  test  team’s 
primary  focus,  specifically  the  OA/DM, 
is  to  ensure  that  test  data  is  reduced  and 
reported  each  day. 

Deviations  from  Test  Plans 

If  the  test  team  believes  that  a  deviation 
from  the  Test  Plan  is  required  during  the 
Pilot  Test  or  Record  Test,  then  the  test 
team  must 

♦  Identify  the  deviation  from  the  plan 

♦  Identify  the  effect  of  the  deviation 

♦  Formulate  in  writing  an  alternate  plan,  or 
document  proposed  changes  to  the  existing 
plan 

♦  Obtain  approval  for  the  changes  before 
execution  from  the  MCOTEA  Division 
Head 


Posttest  Activities 

Failure  Definition /Scoring 
Criteria  Conference 

MCOTEA  convenes  the  FD/SC  Scoring 
Conference  after  the  Record  Test  has 
ended  and  before  the  test  team  leaves 
the  test  site.  MCOTEA,  DC,  CD&I, 
and  the  Program  Manager  each  provide 
a  representative  to  the  conference;  the 
OTPO  represents  MCOTEA  and  serves 
as  chair.  (The  OTPO  may  also  schedule 
intermediate  scoring  conferences  during 
the  Record  Test,  especially  during  a  long 
test  or  one  with  many  TIRs.)  Scoring 
Conference  participants  use  the  guidance 
contained  in  the  system’s  FD/SC  Charter, 
which  was  developed  early  in  the  test 
planning  process.  The  conference  members 
review,  classify,  and  then  vote  on  the 
scoring  of  all  TIRs.  Each  member  has  one 
vote,  but  the  Director,  MCOTEA  casts 
the  deciding  vote  in  the  case  of  a  tie.  The 
scored  TIRs  support  evaluation  of  RAM. 
The  OTPO  should  ensure  the  nearby 
presence  of  essential  personnel  to  respond 
to  questions  or  to  clarify  TIRs.  In  addition, 
the  OTPO  ensures  that  the  following  is 
available  for  the  conference: 

♦  MOIC  of  the  Operating  Forces  for  pre¬ 
scored  TIRs  and  comments  regarding  them 

♦  A  summary  of  TIRs  for  each  member  of  the 
conference.  (Conference  members  should 
review  each  TIR  to  date  and  determine  a 
preliminary  score  before  the  conference 
begins) 

♦  A  summary  of  maintenance  and  times  (start 
time,  stop  time,  and  maintenance  time) 

♦  Copies  of  the  FD/SC  Charter 

♦  System  description,  system  mission,  mission 
time,  crew  correctable  maintenance  actions, 
and  mef  definitions 

Conference  members  score  and  classify 
the  TIRs  by  examining  the  circumstances 
surrounding  each  test  incident  and 
deciding  the  classification,  chargeability, 
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and  hazard/risk  assessment  for  each 
incident.  Refer  to  chapters  3-2  and  6  for  a 
detailed  list  of  these  categories. 

Scoring  is  decided  by  a  simple  majority 
votes.  Incidents  may  be  left  unscored  until 
additional  information  becomes  available 
to  support  a  scoring  decision.  Previously 
scored  incidents  may  be  re-examined 
to  consider  additional  information  if  a 
majority  of  the  conference  members  agrees. 

The  OTPO  documents  the  results  of  the 
Scoring  Conference  in  the  minutes.  Before 
the  conference  concludes,  each  voting 
member  reviews  and  signs  the  minutes. 

Any  conference  member  may  provide  a 
written  dissenting  opinion  on  any  incident 
scoring  result.  The  OTPO  must  include 
any  dissenting  opinions  in  the  conference 
minutes  and  forward  the  signed  minutes  to 
the  Director,  MCOTEA  for  signature. 

Developmental  contractors  are  prohibited 
from  being  involved  in  any  way  in  the 
performance  assessment  or  evaluation 
activities  of  an  operational  test  (OT&E 
of  Defense  Acquisition  Program  2008). 
Accordingly,  developmental  contractors  are 
not  invited  into  the  Scoring  Conferences 
as  observers  or  participants.  However, 
developmental  contractors  can  be  requested 
to  present  information  concerning  system 
design  or  intended  implementation 
procedures,  but  they  must  leave 


immediately  after  providing  information 
or  answering  any  questions  and  before 
further  discussion  ofTIRs  ensues.  Only 
the  Director,  MCOTEA  may  release 
operational  test  data,  including  Scoring 
Conference  results.  Conference  members 
may  not  disclose  any  details  of  the  Scoring 
Conferences  without  the  Director’s 
approval. 

In-Process  Review 

The  In-Process  Review  (IPR)  is  a  meeting 
held  to  provide  early  approval  and  guidance 
to  the  test  team,  specifically  the  OA,  on  the 
adequacy  and  accuracy  of  the  data  analysis. 
The  IPR  occurs  after  the  completion  of 
OT  data  analysis  and  the  FD/SC  Scoring 
Conference.  The  Scientific  Advisor  leads 
a  panel  that  includes  the  S-2  Decision 
Sciences  Lead,  the  Chief  of  Test,  and 
the  Division  Head  for  the  Test  Division. 

All  members  of  the  test  team  involved 
in  preparation  of  the  Test  Data  Report 
should  attend  the  IPR,  which  provides 
an  opportunity  for  the  panel  members  to 
discuss  their  concerns,  investigate  raw  test 
data,  and  review  analytical  methods.  All 
issues  related  to  data  analysis  or  analytical 
methods  must  be  resolved  before  reporting 
final  Measure  results,  which  can  begin  at 
the  IPR’s  end. 
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Step  5:  Operational  Test  Data  Reporting 


Using  MCOTEA’s  standard  Test  Data 
Report  template,  the  OTPO/TM  write 
a  Test  Data  Report  during  posttest 
activities.  The  report’s  purpose  is  to  record 
any  deviations  from  the  Test  Plan  and  to 
package  the  data  for  an  early,  unanalyzed 
look.  The  report  does  not  evaluate  the 
results  or  reach  conclusions  about  OE,  OS, 
and  OSur. 

The  Results  paragraph  advises  the  reader 
to  look  for  information  about  test  data  in 
Annex  B,  presented  by  Measure.  The  data,  if 
copious,  does  not  need  to  be  printed  and  can 
be  attached  to  the  report  on  a  CD. 

The  Test  Data  Report  is  signed  by  the 
Director,  MCOTEA  and  sent  to  DOT&E 
for  programs  on  oversight.  Otherwise  the 
report  is  released  solely  at  the  discretion  of 
the  Director,  MCOTEA. 


Fig.  3-5-1. 
Test  Data  Report  Outline 


Test  Data  Report 
[System  Name] 

1.  Purpose.  This  Test  Data  Report  provides 
raw  and  reduced  test  results  from  the  [type  of 
test]  of  [the  system]  for  an  early,  unanalyzed 
look  at  test  data. 

2.  Background.  MCOTEA  collected  the  data 
in  this  report  in  accordance  the  [type  of]  Test 
Plan  (ref.  a). 

3.  Scope.  This  report  is  limited  to  data  from 
the  test  MCOTEA  conducted  on  the  system  in 
[location]  from  [dates]  . 

4.  Objective.  The  objective  of  this  report  is 
to  make  test  data  available  for  review  while 
MCOTEA  continues  the  evaluative  process 
that  will  lead  to  conclusions  about  Operational 
Effectiveness,  Operational  Suitability,  and 
Operational  Survivability. 

5.  Deviations.  [Summarize  deviations  from 
the  Test  Plan.  Ensure  that  any  deviation  that 
affects  a  data  element  or  data  set  is  explained 
as  a  caveat  to  the  data.  Explain  deviations  and 
caveats  in  detail  in  Annex  A.] 

6.  Methods.  This  report  presents  test  data  in 
electronic  format.  [Assumes  use  of  CD  for  all 
data.  Adjust  if  necessary.] 

7.  Results.  Annex  B  on  the  attached  CD 
presents  a  detailed  breakdown  by  Measure,  in 
tabular  format,  of  the  data  obtained  at  IOT. 

An  index  tab  provides  a  link  to  each  labeled 
Measure. 

8.  References 

a.  MCOTEA.  [ Name  of  Test  Plan.]  [Month 
Year], 


Annex  A.  Test  Plan  Deviations 

Annex  B.  Supporting  Data  for  Test  Measures 


3-5-2 


System  Evaluation 
and  Reporting 


Chapter  3-6 


Step  6:  System  Evaluation  and  Reporting 


Purpose  of  Evaluation 

The  purpose  of  system  evaluation  is  to  answer 
the  evaluation  questions  (i.e.,  Issues,  COIs, 
and  OE/OS/OSur)  contained  in  the  SEP, 
thereby  providing  information  to  decision 
makers  and  PMs  useful  to  system  design  and 
tradeoff  decisions.  The  necessary  input  for 
system  evaluation  is  one  or  more  Test  Data 
Reports,  which  should  naturally  flow  from 
test  events  specified  in  the  TEMP.  The  OA 
is  charged  with  leading  the  evaluation. 

Evaluation  should  begin  at  the  lowest  levels 
of  indenture  (generally  the  Subtask  level) 
at  the  early  stages  of  system  development. 
Little  benefit  exists  in 
delaying  evaluation 
and  reporting  results 
late  in  the  program. 

As  the  system  matures, 
the  evaluations  should 
progress  to  higher 
levels  of  indenture 
until  reaching  the  top 
level  of  the  hierarchy, 
answering  COIs  and 
determining  OE/OS/OSur. 

Evaluation  and  Reporting 
Requirements  for  OT 

After  all  developmental,  live  fire,  and 
operational  testing  is  complete,  the  OA 
leads  the  evaluation  effort  by  using  all 
available  test  data  and  test  reports  to 
complete  the  system  evaluation.  The 
MCOTEA  test  team  plays  a  key  role  in 
the  evaluation  by  providing  contextual 
information,  explaining  any  unusual 
behavior  in  the  data,  and  providing  any 
other  background  information  pertaining 
to  data  taken  during  any  Intermediate 
Assessments,  Operational  Assessments, 
and  IOT.  The  goal  of  the  test  team  at  this 
stage  is  to  help  the  OA  understand  the 
conditions  under  which  individual  tests 
were  conducted  and  data  was  gathered. 


The  evaluation  is  designed  to  accomplish 
the  following: 

♦  determine  if  thresholds  in  the  approved 
capabilities  documentation  and  COIs  have 
been  satisfied 

♦  determine  OE,  OS,  and  OSur  under  realistic 
operational  conditions,  including  Joint  combat 

♦  assess  the  impact  to  combat  operations 

♦  provide  additional  information  on  the 
system’s  operational  capabilities 

As  part  of  the  system  evaluation,  the  OA 
must  include  a  comparison  with  current 
mission  capabilities  using  existing  data  to 
help  determine  measurable 
improvements  brought 
about  by  the  new  system. 
The  cognizant  Test  Division 
will  supply  data  on  current 
mission  capabilities.  If  this 
isn’t  possible,  the  OA  will 
consult  with  the  PM  who 
will  propose  an  alternative 
strategy  for  obtaining  this 
information  (DODI  5000.02  2008).  See 
chapter  3-1  for  details  pertaining  to  each  of 
the  following  process  steps. 

Determine  Threshold  Satisfaction 

The  OA  analyzes  data  from  all  contractor 
DT,  government  DT,  LFT&E,  modeling 
and  simulation,  and  MCOTEA’s 
observations,  assessments,  and  operational 
testing  to  ensure  that  thresholds  have  been 
examined.  The  test  team  determines  which 
thresholds  have  been  met  and  which  have 
not.  The  OER  will  address  both  instances. 

Determine  Operational  Effectiveness 

The  test  team  determines  OE  by 
examining  the  results  of  the  analytic  model 
on  the  COIs.  OE  is  directly  related  to 
mission  effectiveness  and  MCL.  Mission 
effectiveness  is  represented  by  Measures  of 
Effectivness  (MOE).  MOEs  are  typically 


MCOTEA's  evaluation  process 
is  designed  to  be  transparent, 
meaning  that  the  methodology  is 
understandable,  the  data  supports 
the  Measures,  and  the  results  are 
reproducible. 
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associated  with  specific  areas  of  operational 
interest,  each  of  which  contributes  to  the 
system’s  overall  capability  to  accomplish  its 
mission.  OE  can  only  be  determined  as  a 
result  of  operational  testing. 

Determine  Operational  Suitability 

The  evaluator  determines  OS  by  examining 
data  results  from  Measures  of  Suitability 
(MOS)  throughout  program  testing 
and  evaluation.  Areas  of  suitability 
include  but  are  not  limited  to  RAM, 
logistics  supportability,  compatibility, 
interoperability,  training,  human  factors, 
safety,  manpower  and  personnel  selection, 
transportability,  environmental  effects,  and 
system  documentation.  Data  from  many  of 
these  areas  can  be  accumulated  from  early 
program  phases,  and  when  evaluated  with 
OT  data,  helps  determine  OS. 

Determine  Operational  Survivability 

The  test  team  uses  results  from  any 
LFT&E,  IA,  and  CBRN  events 
complemented  by  data  from  a  modeling 
and  simulation  environment  in  conjunction 
with  DT  and  OT  data  to  determine 
OSur.  During  OT,  the  system’s  capability, 
or  lack  thereof,  is  demonstrated  using 
representative  tactics  and  countermeasures 
for  both  friendly  and  opposing  forces. 

The  focus  of  the  OSur  evaluation  is  on 
the  capability  of  the  system  and  the  crew 
to  avoid  damage,  withstand  attack,  and 
recover  capability  in  a  hostile  combat 
environment  without  adversely  affecting 
mission  accomplishment. 

For  information  systems,  the  OSur  evaluation 
examines  information  and  data  security. 

Assess  Impact  to  Combat  Operations 

This  part  of  the  evaluation  examines  how  the 
system  under  test  contributes  to  the  overall 
ability  of  the  Marine  Corps  to  conduct  combat 
operations.  This  assessment,  conducted  by  the 
test  team,  may  be  qualitative  or  quantitative, 
and  the  impact  may  be  small  or  large.  All 
assessments  are  supported  by  data. 


Major  System  Deficiencies 

Major  System  Deficiencies  are  directly 
related  to  the  system  under  test  and  are 
generally  the  failure  of  the  system  to 
attain  a  required  system  capability  or 
attain  a  required  threshold  value  as  stated 
in  the  capabilities  documentation.  These 
deficiencies  are  identified  during  IOT, 
although  the  potential  for  a  Major  System 
Deficiency  may  be  identified  during 
Integrated  Testing.  MCOTEA  notifies  the 
PM  of  any  potential  system  deficiencies 
identified  during  Integrated  Testing. 

Operational  Deficiencies 

During  integrated  testing  and  IOT,  test 
personnel  may  identify  issues  that  affect  the 
performance  of  the  system  under  test,  even 
though  these  issues  cannot  be  associated 
with  a  specific  capability  of  the  system 
under  test.  Indeed,  operational  testing  may  be 
the  first  opportunity  to  discover  these  issues. 

Operational  Deficiencies  tend  to  pertain  to 
interfaces  with  other  systems  or  to  system 
interactions  with  the  Operating  Forces.  In 
some  cases,  these  deficiencies  may  actually 
be  materiel  gaps  in  operational  capability, 
and  in  other  cases,  they  may  illuminate  the 
need  to  create  or  modify  TTPs.  The  test 
team  reports  all  Operational  Deficiencies 
identified  during  any  phase  of  testing. 

Although  Operational  Deficiencies  are  not 
used  in  determining  OE,  OS,  and  OSur, 
if  an  Operational  Deficiency  is  severe 
enough,  MCOTEA  may  recommend  that 
the  system  under  test  not  be  fielded  until 
the  deficiency  is  addressed. 

Types  of  Evaluation  Reports 

Evaluations  that  coincide  with  major 
operational  test  events  are  termed  either 
OTA  Assessment  Reports  (for  EOAs 
and  OAs)  or  OTA  Evaluation  Reports 
(or  OTA  Follow-on  Evaluation  Reports 
(OFER)  (for  IOTs,  MOTs,  and  FOTs). 

All  other  evaluation  reports  published 
before  or  between  major  operational  test 
events  are  termed  Intermediate  Assessment 


All  Major  System 
Deficiencies  identified 
by  MCOTEA  are 
reported.  MCOTEA 
defines  a  major 
deficiency  as 
"A  system  shortfall 
that  adversely  affects 
the  accomplishment 
of  an  operational 
or  mission  essential 
capability  and  no 
known  workaround  is 
available." 
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Reports.  Evaluation  reports  for  System 
Assessments  are  termed  SARs. 

MCOTEA  sends  all  evaluation  reports  to  the 
systems  PM  and  MDA.  Major  evaluations 
such  as  OARs  and  OERs  will  often  reference 
IARs  as  supporting  information.  No  limits 
exist  on  the  number  of  evaluation  reports  that 
may  occur  as  the  system  progresses  through 
its  development  cycle. 

Evaluation  Process  Basics 

The  evaluation  process  is  relatively 
straightforward  because  the  standards 
needed  for  evaluation  were  developed 
in  the  SEP.  The  process,  regardless  of 
the  testing  source  (developmental  or 
operational),  fundamentally  compares  test 
results  with  established  standards. 

Populations  and  Samples 

Before  beginning  the  comparison  it  is 
important  to  understand  the  difference 
between  the  population  and  the  sample. 
Populations,  such  as  the  total  number 
of  helmets  in  the  Marine  Corps,  are 
often  extremely  large  and  represent  the 
entire  universe  of  objects  to  be  evaluated. 

A  population’s  size  usually  makes  it 
impossible,  for  reasons  of  cost  and 
practicality,  to  measure  every  element  of 
the  population.  The  solution  is  to  draw 
samples  from  the  population  and  to 
generalize  from  the  sample  (inference)  to 
the  broader  population. 

Parameters  and  Statistics 

Coinciding  with  the  concepts  of 
populations  and  samples  are  the  concepts 
of  parameters  and  statistics.  A  parameter  is 
any  characteristic  of  the  population,  while 
a  statistic  is  a  characteristic  of  the  sample. 
The  parameters  evaluated  for  a  system 
under  test  are  compared  with  standards 
derived,  in  part,  from  capability  documents. 
Put  another  way,  the  standards  are  what  is 
desired  of  the  population  of  systems  once 
fielded. 

Statistics  are  used  in  the  evaluation  to 


estimate  the  value  of  the  parameter. 

When  testing  is  conducted,  data,  using 
samples,  is  collected,  and  from  that  sample 
statistics  are  calculated.  The  statistic  is  then 
compared  with  the  standard  to  determine  if 
expectations  for  the  population  are  met. 

Considering  Uncertainty  with 
Confidence  Bounds 

Comparing  results  with  standards  should 
always  account  for  uncertainty  to  ensure 
that  the  decision  maker  is  accurately 
informed.  The  evaluator  is  usually 
interested  in  using  the  sample  collected 
during  testing  as  an  estimate  of  the  true 
value  in  the  population.  An  approach  used 
to  assess  for  accuracy  of  the  sample  is  to 
calculate  boundaries  within  which  the  true 
value  is  likely  to  fall.  Such  boundaries  are 
called  confidence  bounds  (Fields  2005). 
When  comparing  the  sample  result  from 
the  test  reports  with  the  standard,  the 
evaluator  should  compare  the  pre-specified 
confidence  bound.  The  confidence  bound 
takes  into  account  the  statistical  error  and 
random  chance  inherent  in  the  testing  to 
express  to  the  decision  maker  how  certain 
the  evaluator  is  about  the  answer. 

Evaluation  at  Early  Stages 
of  System  Development 

Early  test  results  generally  derive  from 
developmental  testing  designed  to  satisfy 
one  or  more  of  the  Issues  at  the  lowest 
levels  of  indenture  in  the  Evaluation 
Framework,  typically  at  the  Subtask  level 
and  below.  The  process  of  evaluating 
these  early  results  begins  with  receipt  of  a 
Developmental  Test  report. 

At  this  early  stage  of  evaluation,  aggregation 
at  the  lower  levels  (Subtasks  and  Tasks)  up 
to  mission  accomplishment  is  not  necessary. 
However,  if  shortfalls  are  identified  in 
evaluation  results  it  is  appropriate  to  identify 
the  potential  ramifications  to  the  next  level 
up  in  the  hierarchy.  In  the  example,  the 
effect  of  the  shortfall  is  undetermined  at 
this  time.  The  evaluation  result,  however,  has 
value  despite  the  fact  that  the  nature  and 
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direction  of  the  cause/effect  relationship  has 
not  been  firmly  established.  The  evaluation 
results  at  this  stage  identify  an  area  of  risk 
that  could  ultimately  have  a  negative  effect 
on  mission  accomplishment.  Figure  3-6- 
1  illustrates  the  linkage,  by  Tasks,  of  the 
trigger  pull  requirement  to  the  outcome  of 
the  sniper  mission. 

The  example  below  is  also  a  proxy 
measurement  being  used  to  satisfy  a 
Subtask.  This  proxy  Measure,  pounds 
of  pull,  falls  squarely  in  the  category  of 
developmental  testing.  This  example 
also  illustrates  integrated  developmental 
and  operational  testing.  In  the  example, 
developmental  testing  has  been  integrated 
in  the  independent  evaluation  of  the  XYZ 
system  by  satisfying  the  information  needs 


for  Subtask  accomplishment  and  threshold 
satisfaction. 

If  the  shortfall  in  performance,  such  as 
in  the  trigger  pull  example,  is  accepted 
by  the  MDA  without  employing  a  fix  or 
mitigation  strategy,  then  the  evaluation  of 
this  capability  is  concluded  and  the  answer 
stands  “as  is.”  Should  fixes  be  required  of 
the  system,  then  retesting  is  required  to 
ensure  that  the  fix  was  successful  and  that 
the  modification  has  not  affected  other 
functions.  In  software  testing  this  is  called 
regression  testing.  Retesting  and  regression 
testing  may  require  updates  to  the  TEMP, 
depending  on  the  program’s  need  to  modify 
schedule,  test  events,  and  resource  changes. 


Figure  3-  6-1. 
Example  of 
Linking  Subtasks 
to  Tasks 


Example  of  Evaluating  at  Early  Stages 

This  example  illustrates  test  data,  test  statistics,  and 
subsequent  evaluation. 

Subtask 

1-X.X  Is  the  trigger  pull  less  than  or  equal  to  4  pounds? 

Measure:  Trigger  Pull  (pounds) 

Threshold:  <  4 

Test  Data  and  Test  Statistics 


Trigger  Pull  (pounds) 

Weapon 

Weapon  1 

Weapon  2 

Weapon  3 

1 

4.40 

5.22 

4.90 

2 

3.76 

5.08 

4.63 

3 

Trial 

4 

4.37 

3.92 

3.61 

5.39 

5.05 

3.22 

5 

5.91 

4.65 

3.32 

6 

4.86 

3.80 

5.99 

7 

4.16 

5.45 

4.99 

8 

3.87 

4.65 

4.93 

9 

4.09 

5.98 

3.26 

10 

4.15 

5.02 

4.93 

The  developmental  test  results  in  the  tables  to  the  right  were  documented  in 
the  test  report  of  the  XYZ  system. 


Evaluation  of  Test  Results  in  the  IAR 

1-X.X  Is  the  trigger  pull  less  than  or  equal  to  4  pounds?  Answer:  No. 

According  to  the  rationale  for  the  requirement  in  the 
XYZ  Capability  Production  Document,  “The  XAZ’s 
trigger  pull  should  be  light  enough  to  allow  for  precision 
engagements,  yet  provide  enough  resistance  to  be  safely 
employed  in  a  combat  environment.”  Based  on  the  sample  data  from  the  developmental  testing,  MCOTEA  is  95 
percent  confident  that  the  true  population  mean  for  trigger  pull  is  at  least  4.82  pounds  or  less.  Because  the  lower 
confidence  bound  is  greater  than  the  threshold  value  of  4.00,  MCOTEA  has  sufficient  certainty  to  conclude  that 
the  requirement  has  not  been  satisfied.  The  potential  exists  for  this  requirement’s  shortfall  to  have  a  negative  effect 
on  the  target  engagement  task  performed  by  the  sniper.  It  is  possible  that  the  shortfall  may  manifest  itself  as  a 
causal  factor  in  reduced  probability  of  hit,  which  is  the  Measure  of  Performance  for  target  engagement.  Overall 
impact  of  this  shortfall  on  Operational  Effectiveness  cannot  be  evaluated  at  this  time. 


Statistics 

Mean 

4.58 

Standard  Deviation 

0.78 

Standard  Error 

0.14 

alpha 

0.05 

tipper  Confidence  Bound 

4.82 
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MCOTEA  performs  the 
evaluation  process 
consistently  the  same 
across  the  evaluation 
continuum.  The 
products  of  evaluation 
are  either  assessment 
reports  or  evaluation 
reports,  depending 
on  the  type  of  test 
event  that  yielded 
the  data  and  whether 
the  system  requires 
operational  testing 
or  not.  Simply  stated, 
evaluation  of  any  test 
event  short  oflOT 
produces  a  SAR,  an  IAR, 
or  an  OAR;  evaluation 
of  any  test  event  that 
is  I0T  and  beyond 
produces  an  OER. 


Evaluation  at  Later  Stages 
of  System  Development 

At  later  stages  of  system  development, 
evaluation  addresses  Issues  at  the  Task 
level  or  suitability  Issues  at  the  major 
component  or  system  level. 

As  a  system  progresses  in  its  development, 
MCOTEA  may  perform  an  Operational 
Assessment  as  a  pre-IOT  event.  (An 
Early  Operational  Assessment  is  also 
possible.)  In  evaluating  the  results  of 
an  OA,  MCOTEA  does  not  determine 
OE/OS/OSur. 

Evaluations  before  IOT  generally  do  not 
aggregate  Task-level  information  to  COIs 
and  OE.  However,  these  evaluations  can 
contribute  to  determination  of  OS  and 
OSur.  It  is  not  appropriate  to  report  any 
OE/OS/OSur  conclusions  at  this  stage  for 
the  following  reasons: 

♦  Scheduled  IOT  needs  to  be  performed  and 
entrance  criteria  need  to  be  satisfied 

♦  Premature  reporting  of  OE/OS/OSur  could 
give  the  false  impression  that  the  system 

is  sufficiently  mature,  stable,  and  ready  for 
full-rate  production 

♦  Premature  reporting  could  also  negatively 
affect  program  success  by  labeling  the 
program  deficient  when  in  reality  it 

has  not  had  sufficient  time  to  satisfy  all 
requirements  and  achieve  stability 

As  a  system  moves  into  later  stages  of 
development,  the  evaluation  process 
remains  the  same  but  the  source  of  testing 
may  change,  depending  on  the  evaluation 
question.  In  the  example  on  the  next  page, 
Marine  operators  are  using  the  system  on 
test  ranges.  The  emphasis  is  on  threshold 
conditions  for  Task  performance. 

Evaluating  for  OE/OS/OSur 

MCOTEA  determines  OE/OS/OSur  by 
evaluating  screening  criteria  and  COIs, 
only  after  specific  operational  test  events 
have  occurred  that  generate  the  remaining 
required  test  data. 


Evaluating  Screening  Criteria 

Evaluating  screening  criteria  is  the  first 
step  in  determining  OE/OS/OSur.  As 
discussed  in  chapter  3-1,  screening  criteria 
can  simplify  the  evaluation  process  by 
reducing  the  number  of  evaluation  criteria 
to  only  those  essential  for  determining 
worth  or  value.  (A  system  that  fails  to 
meet  minimum  screening  criteria  should 
not  proceed  to  final  evaluation.)  Screening 
criteria  can  be  thought  of  as  gates  that  force 
evaluations  to  occur  in  steps  as  the  system 
matures  or  information  becomes  available. 

OS  presents  a  particular  and  significant 
challenge  to  the  system  evaluator  because 
the  topics  contained  in  OS  are  numerous 
and  diverse.  For  example,  transportability 
can  be  a  binding  constraint  for  OS  and  can 
lead  to  a  determination  of  Not  OS: 

One  of  the  Tasks  within  system  X’s  mission 
is  to  externally  transport  the  system  by 
CH-53E  helicopter.  However,  the  system 
failed  the  lift  testing  and  cannot  be  certified 
for  that  mode  of  transport.  Given  the 
importance  of  this  certification  for  this  mode 
of  transport,  this  system  is  considered  to  be 
Not  Operationally  Suitable. 

A  second  example  illustrates  a  different 
aspect  of  screening  criteria  used  to 
constrain  the  evaluation’s  outcome.  Safety 
is  a  necessary  trait  for  any  system  fielded 
for  use  by  Marines  and  is  a  consideration 
under  OS: 

During  testing  of  system  X,  MCOTEA 
discovered  a  safety  hazard  that  was  scored 
with  a  severity  of  “catastrophic”  and  a 
probability  of  “probable”  based  on  the  safety 
evaluation  scoring  matrix  developed  from 
MIL-STD-882D.This  safety  hazard  falls 
into  the  red  zone  of  safety  scoring,  meaning 
that  the  hazard  is  of  a  very  serious  nature. 
Consequently,  MCOTEA  determines  the 
system  to  be  Not  Operationally  Suitable 
until  the  hazard  is  mitigated. 


3-6-6 


Evaluate  Test  Results 


Evaluating  COIs 

After  evaluating  screening  criteria,  the  OA 
inserts  test  data  into  the  analytic  model 
developed  specifically  for  the  COIs.  The 
principal  task  here  is  to  use  the  analytic 
model  from  the  SEP  to  compare  against  the 
threshold  for  effect.  The  following  example 
of  an  IOT  performed  on  the  ABC  Attack 
System  assumes  that  all  earlier  stages  of 
evaluations  have  been  successfully  completed 
and  the  screening  criteria  have  been  satisfied. 


The  ABC  Attack  System  provides 
electronic  attack  equipment  capable  of 
detecting  and  targeting  an  adversary’s 
communications  across  the  spectrum  of 
contemporary  MAGTF  operations.  In 
this  example  the  ABC  Attack  System 
is  replacing  a  legacy  system  having  the 
same  mission  and  achieving  the  same 
effect,  albeit  at  lower  levels  than  the  ABC 
Attack  System’s  requirements.  Given  this 
information,  the  ABC  Attack  System  has  a 
single  COI  and  MOE  (seen  on  next  page). 
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Example  of  Evaluating  at  Later  Stages 

This  example  illustrates  the  importance  of  threshold  conditions  in  conjunction  with  performance. 
Task 

l-X.  Can  the  operators  using  the  XYZ  system  successfully  engage  targets  with  a  probability  of 
0.70? 

Measure:  Probability  of  hit 
Threshold:  >  0.70 
Test  Data 

The  operational  test  results  in  the  tables  are  from  a  fictional  Operational  Assessment  of  the  XYZ 
system. 

Test  Statistics 
Evaluation  of  Test  Results 

l-X.  Can  the  operators  using  the  XYZ  system  successfully  engage  targets  with  a  probability  of 
0.70?  Answer:  No. 

According  to  the  rationale  for  the  requirement  in  the  XYZ  Capability  Production 
Document,  “The  XYZ  shall  have  the  ability  to  precisely  engage  targets  at  long 
range  with  a  high  probability  of  first-round  lethal  hit.  This  will  enhance  the 
operator’s  ability  to  carry  out  operations  and  inflict  damage  on  enemy  forces  at 
longer  ranges  than  current  semiautomatic  sniper  rifles  can  achieve  within  the 
current  inventory  while  augmenting  the 


capabilities  of  the  M40A3.”  Based  on 
the  sample  data  from  the  developmental 
testing,  MCOTEA  is  95  percent 
confident  that  the  true  probability  of 
hit  is  at  least  0.67.  Because  the  lower 
confidence  bound  is  less  than  the 

threshold  value  of  0.70,  MCOTEA  cannot  conclude  that  the  requirement 
has  been  satisfied  under  all  operating  conditions.  Based  on  the  sample  data, 
temperature  conditions  appear  to  be  a  significant  predictor  of  probability  of 
hit.  In  nominal  and  hot  environmental  conditions  the  probability  of  hit  is 
adequate  to  satisfy  the  requirement;  however,  it  drops  significantly  in  cold 
environments.  This  reduction  in  probability  of  hit  in  cold  environments  limits 
the  effectiveness  of  the  sniper  mission  by  reducing  the  operating  environments 
to  nominal  and  hot. 


Statistics 


Probablity  of  Hit 

0.76 

alpha 

0.05 

Lower  Confidence  Bound 

0.67 

p-value 

0.15 

Cold  (-25°to  -5  °F) 

0.57 

Nominal  (60°  to  80  °F)* 

0.90 

Hot  (110°  to  125  °F)** 

0.80 

‘Significant  Finding  (p=0.007) 

“Marginally  Significant  Finding  (p=0.057) 
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TheCOI  result  of 
0.42  is  based  on 
the  data  obtained 
during  IOT  as  well  as 
accumulated  data  from 
integrated  testing. 
When  necessary, 
those  data  can  also 
be  supplemented  by 
the  results  of  properly 
accredited  models  and 
simulations. 


COI-1.  Can  the  Marines  using  the  ABC 
Attack  System  achieve  a  probability  of 
Jamming  Mission  Success  (P  )  of  at 
least  0.52? 

MOE-1.  Probability  of  jamming  mission 
success. 

This  chapter  uses  the  same  example  shown 
in  chapter  3-1: 

P  .  =  P.  x  P  /  x  P.  x  R  x  A 

mission  d  a  d  j  o 

The  components  of  the  analytic  model  that 
were  sourced  from  the  ABC  Attack  System 
Capability  Development  Document  are  as 
follows: 

♦  Probability  of  Detection  (P  ) 

♦  Probability  of  Electronic  Attack  given  a 
Detection  (Pa|d) 

♦  Probability  of  Jamming  (Frequency  Range 
20  -  2500  MHz  (P.)) 

♦  Reliability  (R) 

♦  Availability  (AJ 

After  inserting  threshold  values  into  the 
analytic  model,  the  threshold  value  for 
P  .  was  calculated  to  be  0.52. 

mission 

Testing  results  are  depicted  in  the  next 


Parameter 

Statistics 

0.98 

Pa/d 

1.00 

P 

J 

0.79 

R 

0.78 

Mission  Duration  (hours) 

24 

MTB0MF  (hours) 

99 

A 

0 

0.69 

MTBM  (hours) 

56 

MDT  (hours) 

25.1 

MTTR  (hours) 

1.1 

*MLDT  (hours) 

24 

^Source  from  Independent  Logistic  Support  Plan 

column. 

Using  the  analytic  model  and  the  tested 
results,  the  Probability  of  Mission  Success 
(P  )  is  equal  to  0.42,  as  seen  below: 

v  mission'  -i  ' 

P  .  .  =  .98  x  1.00  x  0.79  x  0.78  x  0.69  =  0.42 

mission 

The  0.42  value  represents  the  COI  result. 

The  example  illustrates  that  the  system 
failed  to  achieve  the  minimum  level  of 
performance  expected  for  the  system 
because  the  P  .  .  is  less  than  the  required 
0.52  (as  derived  from  the  thresholds  in  the 
capabilities  documentation).  This  example 


Bias  in  Parameters 

Operational  test  is  a  valuable  opportunity  for  obtaining  realistic  estimates  for  parameters,  but  not  necessarily  all 
parameters. 

Operational  Availability  (Ao)  is  a  parameter  that  usually  carries  significant  bias  in  the  operational  test.  Ao  is  a 
function  of  the  intervals  required  for  maintenance  and  the  time  to  maintain  the  system.  Time  to  maintain  is  biased 
because  maintainers  and  parts  are  co-located  with  the  test  unit. 

Co-location  from  a  test  perspective  is  sound  because  it  enables  greater  system  availability  during  OT.  However, 
co-location  also  creates  the  detrimental  effect  of  inflating  Ao.  In  reality,  parts  and  maintainers  are  often  separated 
by  time  and  distance,  which  delays  repairs  and  reduces  the  Ao. 

In  the  sample  test  for  the  ABC  Attack  System  in  this  section,  the  Ao  for  the  system  as  observed  in  OT  is  0.98  because 
essentially  no  logistics  down  time  occurred.  However,  the  Independent  Logistics  Support  Plan  cites  an  average 
estimate  of  24  hours  for  mean  logistics  down  time,  indicating  that  the  test  number  for  Ao  is  inflated. 

The  ultimate  effect  of  calculating  P  .  .  for  this  system  without  considering  logistics  down  time  is  seen  in  the 
following  equation,  where  the  calculated  P  . .  is  inflated  from  0.42  to  0.60  when  using  the  A  observed  during  I0T. 

'  '  mission  -'o 

P  .  .  =  .98  x  1 .00  x  0.79  x  0.78  *  0.98  =  0.60 
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has  a  single  COI,  but  other  systems  may 
have  more  than  one.  To  address  why  the 
system  is  underperforming,  the  evaluator 
first  calculates  the  MCL  for  COIs  and 
then  analyzes  the  test  results. 

Calculating  MCLs  for  COIs 

Using  the  COI  results  from  the  analytic 
models,  the  evaluator  calculates  the  MCL 
for  each  COI  using  the  decision  model 
from  the  SEP.  The  output  of  this  process 
is  a  value  on  the  MCL  scale  between  0  and 
100  for  each  COL 

To  find  MCL,  in  this  example  a  piecewise 
linear  function  (below)  depicts  the  last  two 
rows  of  data  from  the  table  below.  The  table 
contains  results  from  the  analytic  model 
with  P  on  the  x-axis  and  MCL  on  the 

mission 

y-axis. 

This  process  must  be  repeated  for  each 


Probability  of  Jamming  Mission  Success  (Xi) 


Parameter  Lowest  Current  Threshold  Objective 

Possible  Capability  Capability  Capability 

Pd 

0.00 

0.30 

0.80 

0.95 

Pa/d 

0.00 

1.00 

1.00 

1.00 

p; 

0.00 

0.80 

0.80 

0.80 

R 

0.00 

0.90 

0.90 

0.90 

A 

0 

0.00 

0.90 

0.90 

0.90 

P  . .  (x-axis) 

0.00 

0.19 

0.52 

0.62 

MCL  (y-axis) 

0 

50 

80 

100 

COI  if  multiple  COIs  exist  in  the  SEP. 

Analysis  of  COI  Results 

Each  parameter  estimate  obtained  from 
testing  has  some  level  of  error  associated 
with  it.  In  fact,  using  the  design  of 
experiments  process  allows  for  systematic 
variation  of  factors  of  interest  for  the 
express  purpose  of  determining  if  the 


factors  significantly  influence  the  outcome. 
The  value  of  0.42  represents  a  point  from 
which  to  score  the  system  in  total,  but 
important  information  for  the  decision 
maker  is  nested  within  the  statistics  used  to 
generate  that  point  estimate  for  effect. 

In  the  ABC  Attack  System  example,  the 
system  was  operated  during  the  operational 
test  in  both  stationary  and  on-the-move 
modes.  Each  mode  had  20  planned  trials 
for  a  total  of  40.  The  summary  statistics 
from  the  Test  Report  indicate  that  39 


Statistics  for  P  Overall 

Stationary  On-the-Move 

Successes 

31 

16 

IS 

Total  Attempts 

39 

19 

20 

Point  Estimate 

0./9 

0.84 

0.75 

Confidence  (alpha  =  0.05) 

One-sided  Lower  Bound 

0.66 

N/A 

N/A 

Two-sided  Lower  Bound 

N/A 

0.60 

0.51 

Two-sided  Upper  Bound 

N/A 

0.97 

0.91 

valid  trials  were  collected;  the  results  are 
displayed  in  the  table  below. 

Analysis  of  the  data  indicates  that  a 
shortfall  exists  in  jamming  performance. 

The  evaluator  can  conclude  that  the  P.  for 
the  population  is  at  least  0.66,  which  is 
insufficient  certainty  for  concluding  that 
the  threshold  of  0.80  for  this  parameter  has 
been  satisfied.  In  addition,  the  two  modes  of 
operation  do  not  appear  to  be  significantly 
different  in  terms  of  jamming  success. 

Finally,  mode  of  operation  by  itself  does 
not  appear  to  cause  a  practical  change 
in  the  mission  capability.  Even  if  the 
aggregate  value  of  P.  was  replaced  with  a 
very  optimistic  value  of  0.98,  the  calculated 
value  for  P  is  still  below  the  overall 

mission 

required  value  for  P  .  of  0.52.  This 
means  that  improvement  in  probability  of 
jamming  by  itself  will  not  overcome  the 
shortfall.  Improvements  in  other  areas  must 
be  made  to  improve  the  systems  MCL. 

The  example  here  illustrates  only  a  small  part 
of  the  analysis  that  should  occur  to  explain 
the  results  of  the  evaluation.  From  here  the 
evaluator  continues  to  thoroughly  explore 


Purpose  of 
Determining  MCL 

■=>  Provides  a  systematic 
methodology  for  arriving 
atOE,  OS,  and  OSur 
conclusions 

■=>  Allows  the 
aggregation  of  measures 
using  different  units 
by  converting  the 
measurement  results  to 
the  dimensionless  MCL 
value  function 

■=>  Provides  a  framework 
for  aggregation  when 
multiple  COIs  (missions) 
are  an  element  of  the 
evaluation 

■=>  Normalizes  evaluation 
results  to  a  common 
scale  (MCL  score 
between  0-100),  which 
allows  decision  makers 
responsible  for  multiple 
programs  to  assess 
demonstrated  capabilities 
across  their  portfolio  in 
consistent  terms 


3-6-9 


Chapter  3-6 


the  data  results  and  prepares  to  inform  the 
decision  maker  of  the  evaluations  conclusions. 

OE/OS/OSur  Conclusions 

Determining  OE 

The  final  step  in  the  evaluation  is  to  arrive 
at  the  top-level  answers,  i.e.,  determining 
if  the  system  is  OE.  OE  is  the  first  answer 
that  must  be  computed  based  on  the  MCL 
scores  of  the  subordinate  COIs.  Systems 
with  multiple  COIs  must  have  their  answers 
aggregated  using  the  weights  from  the  SEP. 
Although  not  fully  illustrated  here,  the 
process  consists  of  taking  the  MCL  score 
(MCL.)  for  each  COI  and  multiplying  by 
the  COI  weights  (w.)  from  the  SEP. 

The  next  step  is  to  sum  the  weighted  scores 
to  arrive  at  an  aggregate  score  across  all 
missions.  When  the  evaluation  contains  only 
one  COI,  the  weight  defaults  to  100  percent, 
thereby  making  the  MCL  score  the  same  as 
the  aggregate  scores  computed  below.  The 
resultant  score  determines  if  the  system  is 
OE  or  Not  OE  using  the  following  formula: 

n 


i=l 


An  example  set  of  data  for  the  ABC  Attack 
System  can  be  found  in  the  following  table. 


Aggregate  MCL  Score 

Conclusion 

n 

80  <  OE  =  ^  W;  (MCL,)  <  100 
i=l 

OE 

n 

0  <  OE  =  ^  wf  (MCL,)  <  80 
i= 1 

NotOE 

This  notional  example  assumes  the  ABC 
Attack  System  has  three  COIs,  weighted 
at  50  percent,  20  percent,  and  30  percent, 
respectively.  The  MCL  score  for  each  COI 
is  in  the  third  column  while  the  weighted 


COI=/  W.  MCL  w.  (MCL) 

1 

50% 

71 

36 

2 

20% 

54 

10 

3 

30% 

83 

25 

OE=2>.  (MCL) 

71 

MCL  score  is  in  the  fourth. 


In  the  example,  the  overall  weighted 
score  is  71,  translating  into  a  conclusion 
of  Not  OE.  The  shortfalls  can  be  traced 
immediately  to  lower  than  expected 
capabilities  in  COIs-1  and  2.  COI-3  is 
more  than  adequate,  but  given  that  its 
weight  is  only  30  percent  in  the  evaluation, 
its  adequacy  does  not  counteract  the  effect 
of  the  shortfalls  in  the  other  COIs. 

Using  the  previous  example,  the  evaluator  can 
answer  the  OE  evaluation  question  as  follows: 

Is  the  OE  of the  ABC  Attack  System  adequate  to 
achieve  an  average  Mission  Capability  Level  score 
of  at  least  80  out  of 100? 

Answer:  No.  The  MCL  of  the  ABC  Attack 
System  across  all  missions  falls  below  the 
threshold  score  of  80;  however,  the  system 
performed  better  than  the  currently  fielded 
system.  ABC  Attack  System  performs  above 
expectation  on  the  COI-3  mission,  but  below 
expectation  on  the  COI-1  and  COI-2  missions. 

This  simplified  example  applies  the  MCL 
scoring  at  the  COI  level  containing  one 
MOE.  For  COIs  containing  multiple 
MOEs  see  the  discussion  “Applying  the 
MCL  Function  at  the  Next  Level  Down,” 
at  the  end  of  chapter  3-1. 

Determining  OS  and  OSur 

Having  determined  OE,  the  evaluator  now 
determines  OS  and  OSur.  To  simplify  the 
examples,  the  use  of  multiple  COIs  and  the 
determination  of  OSur  are  dropped  here, 
but  the  procedure  for  examining  OSur 
remains  the  same.  Using  the  OE  score  as  a 
point  of  reference,  the  evaluator  determines 
OS  and  OSur  with  the  same  analytic 
model  used  to  determine  OE. 

In  this  continuing  example,  the  evaluator 
already  knows  that  the  system  is  not 
achieving  sufficient  effect,  evidenced  by 
the  MCL  score  of  less  than  80.  The  next 
step  is  to  trace  the  source  of  the  problems 
to  one  or  more  root  causes.  The  specific 
evaluation  questions  are  as  follows: 

Is  the  OS  of  the  XXX  system  adequate  to 
achieve  an  average  MCL  score  of  at  least  80  out 


3-6-10 


Evaluate  Test  Results 


of  100  when  performance  and  survivability  are 
held  constant  at  threshold  levels ? 

Is  the  OSur  of  the  XXX  system  adequate  to 
achieve  an  average  MCL  score  of  at  least  80 
out  of  100  when  Performance  and  Suitability 
are  held  constant  at  threshold  levels ? 

The  data  set  used  to  arrive  at  the  conclusions 
for  OS  and  OSur  is  the  same  as  that  for 
OE.  However,  more  detail  is  required  to 
isolate  the  cause  of  the  shortfall  in  effect  to 
Performance,  Suitability,  and/or  Survivability 
The  evaluator  must  understand  the 
constituent  components  of  the  evaluation 
model  to  arrive  at  causality  The  tables  to 
the  right  illustrate  two  different  sets  of  data. 
The  first  set  is  the  threshold  parameters  for 
the  ABC  Attack  System.  The  second  set 
is  statistics  from  testing  used  to  estimate 
achievement  of  the  thresholds. 

The  Threshold  column  indicates  the 
minimum  acceptable  parameter  values 
from  the  SEP.  The  Tested  Results  column 
represents  the  tested  values  for  all  of  the 
parameters  of  interest.  The  tested  values 
consider  both  Performance  and  OS 
parameters  simultaneously  to  identify  a 
combined  effect. 

However,  not  all  tested  values  meet 
the  minimum  threshold  values.  The 
performance  parameter  P.  and  Os  and 
parameters  R  and  Ao  fell  short  of  their 
respective  thresholds,  raising  the  question, 
“Is  the  MCL  score  of  71  caused  by 
performance  parameters,  OS  parameters, 
or  both?”  To  answer  this,  the  evaluator 
performs  sensitivity  analysis  on  the  results 
to  see  the  parameters’  influence  on  the  OE 
outcome.  To  do  this  the  computations  of  the 
analytic  model  are  redone  by  first  setting  all 
performance  parameters  to  Threshold  values 
and  setting  all  OS  parameters  to  Tested 
Results  (top  table). 

The  result  from  this  sensitivity  analysis 
indicates  that  when  considering  OS  by  itself, 
the  system  falls  below  the  minimum  score  of 
80.  Based  on  this  process  the  evaluator  can 
answer  the  OS  question  this  way: 


Is  the  OS  of  the  ABC  Attack  System  adequate 
to  achieve  an  MCL  score  of  at  least  80  out  of 
100  when  performance  and  survivability  are 
held  constant  at  threshold  levels ? 

Answer:  No.  ABC  Attack  System  scores 
an  MCL  of  65  out  of  100  when  the  OS 
parameters  are  considered  by  themselves  with 
performance  parameters  held  constant  at 
threshold  value .  The  source  of  the  shorfall  in 
MCL  score  can  be  traced  to  both  Reliability 
and  Availability.  Analysis  of  the  results 
indicates  that  ABC  Attack  System  is  not  as 
equally  reliable  under  all  employment  modes. 
Missions  conducted  on-the-move  cause  system 
failures  that  reduce  the  capability  of  the  system. 
Additionally,  the  planned  Logistics  Down 
Time  combined  with  the  failure  rate  has  a 
detrimental  effect  on  system  Availability. 


Parameter  Threshold  Tested 

Results 

pd 

0.80 

0.98 

Pa/d 

1.00 

1.00 

P 

J 

0.80 

0.79 

R 

0.90 

0.78 

A 

O 

0.90 

0.69 

P  .  .  (x-axis) 

0.52 

0.42 

MCL  (y-axis) 

80 

71 

Parameter  Threshold  Tested 

Results 

pd 

0.80 

Pa/d 

1.00 

P. 

J 

0.80 

R 

0.78 

A 

O 

0.69 

P  .  .  (x-axis) 

0.35 

MCL  (y-axis) 

65 
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Sensitivity  Analysis  on 
Performance 

Sensitivity  analysis  on  the  performance 
parameters  is  accomplished  the  same 
way  as  for  OS.  The  evaluator  reruns  the 
computations  of  the  analytic  model  by  first 
setting  all  OS  parameters  to  Threshold  values 
and  setting  all  performance  parameters  to 
Tested  Results.  The  table  below  illustrates  the 
values  needed  and  the  results  of  calculations. 

The  result  from  this  sensitivity  analysis 
indicates  that  when  considering 
performance  parameters  by  themselves, 
the  system  is  adequately  performing  when 
functional  because  the  score  is  100.  From 
this  result,  the  evaluator  can  conclude  that 
when  it  works,  the  system  does  what  is 
expected.  In  addition,  the  shortfall  in  the 
threshold  for  jamming  does  not  significantly 
affect  the  observed  deficient  performance. 

The  shortfall  in  P.  appears  to  be  adequately 
compensated  for  by  the  fact  that  ABC  Attack 
System  detects  with  a  much  greater  likelihood 
than  what  was  expected  of  the  system. 


Parameter 

Threshold 

Tested  Results 

pd 

0.98 

Pa/d 

1.00 

P 

J 

0.79 

R 

0.90 

A 

0 

0.90 

P  . .  (x-axis) 

mission 

0.63 

MCL  (y-axis) 

100 
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Evaluation  Conclusions 
and  Recommendations 

The  preceding  notional  example  illustrated  the 
mechanism  for  deliberately  and  systematically 
arriving  at  a  series  of  evaluation  conclusions. 
This  process,  while  lengthy,  is  sufficiently 
transparent  to  allow  outsiders  to  examine  and 
replicate  results  in  an  independent  setting. 

The  process  is  also  a  useful  tool  for  decision 
makers  and  engineers  when  deciding  on 
improvements  to  current  capabilities. 

The  transparency  of  the  process  and  its 
analytic  nature  also  lend  themselves  to 
future  evaluations  that  might  occur  to 
see  how  the  system  scores  in  relation  to 
expectation,  present  performance,  and 
desired  future  capability. 

The  specific  conclusions  that  can  be  drawn 
from  the  preceding  examples  can  be 
summarized  in  the  following  statement: 

ABC  Attack  System  is  Not  OE  because  the  system 
is  Not  OS.  Overall  the  ABC  Attack  System  is 
better  than  the  currently  fielded  system ,  but  falls 
below  expectation  in  three  of  four  critical  areas. 
The  cause  for  the  reductions  in  effectiveness 
stem  from  less  than  required  Reliability 
and  Availability.  In  particular,  the  systems 
Reliability  in  mission  environments  where  the 
ABC  Attack  System  performs  on-the-move  does 
not  measure  up  to  standards.  The  ABC  Attack 
System  should  not  be  expected  to  achieve  the 
required  effect  in  all  mission  environments,  based 
on  the  identified  shortcomings. 

Any  recommendations  included  in  the 
evaluation  report  should  trace  directly 
from  the  results  and  conclusions.  In 
the  preceding  example  the  logical 
recommendation  would  be  to  improve 
Reliability  of  the  system  while  on- 
the-move.  The  evaluator  should  avoid 
recommending  specific  engineering 
changes  or  solutions.  MCOTEA  personnel 
are  not  charged  with  recommending 
specific  solutions,  no  matter  how 
promising  they  may  be.  MCOTEA’s 
recommendations  should  only  identify 
areas  of  Performance,  Suitability,  and 
Surviviability  that  warrant  improvement, 
based  on  test  results. 
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AnnexAj  Data  Depiction 


Why  Depict? 

MCOTEA  depicts  data  for  several 
reasons — to  create  visual  interest, 
to  support  the  text,  and  of  primary 
importance,  to  explore  the  data.  Graphics 
are  instruments  for  reasoning  about 
quantitative  information.  For  example, 
graphs  can  be  used  to  evaluate  changes  over 
space  or  time,  compare  ideas,  and  provide 
a  tool  for  evidence-based  reasoning.  The 
following  pages  provide  guidance  for 
MCOTEA  staff,  in  particular  OAs,  for 
creating  appropriate  depictions  of  data. 

Exploring  the  Data 

Exploratory  data  analysis  can  maximize 
insight  into  data  sets,  detect  outliers 
and  anomalies,  and  test  underlying 
assumptions.  For  example,  graphing  reveals 
patterns  in  the  data  that  would  not  be 
apparent  from  a  table  or  spreadsheet. 

Further  exploration  can  also  aid  in 
determining  the  distribution  of  the  data, 
which  will  help  to  determine  valid  methods 
of  statistical  analyses.  For  example,  when 
comparing  test  data  against  a  theoretical 
normal  distribution,  graphing,  along  with 
goodness  of  fit  tests,  will  help  determine 
whether  the  data  is  normally  distributed. 

A  graphical  and  statistical  analysis  of 
the  data  distribution  is  also  required  for 
Reliability  equations.  For  example,  if  a 
Reliability  equation  assumes  an  exponential 
distribution,  graphing  (and  goodness  of  fit 
tests)  will  help  validate  that  assumption. 

How  Often  to  Depict? 

A  good  ratio  to  strive  for  in  technical 
documents  is  25  percent  depiction 
(graphs,  tables,  diagrams,  and  images) 
and  75  percent  text.  In  the  main  body  of 
MCOTEA’s  reports,  which  are  targeted  at 
the  05/06-level  audience,  the  graphs  should 
match  the  overall  level  of  the  text.  The 


reports’  technical  annexes  are  appropriate 
for  graphs  that  are  more  analytical,  such  as 
distribution  plots. 

However,  even  these  more  technical  graphs 
should  present  the  data  in  a  manner 
that  allows  the  reader  to  quickly  and 
unambiguously  grasp  what  the  data  means. 

Which  Depictions  to  Use? 

The  type  of  depiction  needed  for  a 
document  depends  on  the  data  and  the 
point  trying  to  be  made.  The  following 
questions  are  helpful  in  trying  to  decide  on 
a  depiction  method: 

♦  Are  categories  of  data  being  compared? 

♦  Are  trends  or  correlations  between  two  or 
more  variables  visible? 

♦  Are  trends  depicted  over  time? 

♦  Is  data  distribution  visible? 

♦  Is  Reliability  data  being  depicted? 

♦  Are  survey  results  being  depicted? 

♦  Are  OE,  OS,  and  OSur  results  being  depicted? 

Types  of  Depiction 

Graphs  are  used  to  display  data  efficiently, 
meaningfully,  and  unambiguously  to 
supplement  and  support  the  text  of  the 
document.  They  reinforce  and  clarify 
the  text  by  telling  a  story  pictorially. 
Distribution  graphs  can  include 
histograms,  line  graphs,  and  probability 
plots.  Fine  graphs  and  histograms  are  easy 
to  read  and  are  helpful  in  depicting  outliers 
and  skewness  as  well  as  the  distribution 
of  the  data.  Probability  plots,  which  can 
include  CFQ_  plots,  are  a  more  powerful 
approach  to  comparing  distributions,  but 
require  more  skill  to  interpret. 

The  information  contained  in  this  section  is 
from  the  NIST/SEMATECH  e-handbook  of 


What  is  to 
be  sought 
in  designs  for 
the  display  of 
information  is  the 
clear  portrayal  of 
complexity.  Not 
the  complication 
of  the  simple; 
rather  the  task 
of  the  designer 
is  to  give  visual 
access  to  the 
subtle  and  the 
difficult — that  is, 
the  revelation  of 
the  complex. 

-Edward  Tufte 
(1996) 
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Microsoft  Excel®  vs.  Word ® 

Pro:  Microsoft  Excel  uses  an 
easier  automated  results 
population  from  SQL  or  other 
databases.  The  user  is  also 
able  to  add  locked-in  drop 
tables  to  reduce  variations. 

Con:  Converting  an  Excel 
spreadsheet  to  .pdf  requires 
reformatting  for  compilation. 
Also,  the  tools  and  capabilities 
within  Excel  are  not  always 
familiar  to  general  users. 

Pro:  Microsoft  Word  tables 
are  more  easily  manipulated 
than  Excel  tables.  Word  also 
provides  the  user  with  a  more 
familiar  toolset.  Converting 
documents  to  .pdf  does  not 
require  reformatting. 

Con:  Microsoft  Word  does  not 
offer  automatic  population 
for  tables,  which  could  lead  to 
version  control  issues.  Word 
also  provides  limited  formula 
support  and  no  drop  table 
support. 

Conclusion:  when  data  will  be 
kept  solely  on  a  CD  (no  hard 
copy  required),  Excel  may 
be  the  better  choice.  When 
lengthy  data  must  be  printed, 
Word  may  be  the  better 
choice. 


Statistical  Methods  (2010).  General  guidelines 

for  graphing  are  as  follows: 

♦  Histograms  are  similar  to  bar  graphs 
except  each  bar  represents  numbers  that  are 
grouped  and  form  a  continuous  range  from 
left  to  right.  Histograms  can  be  used  to 
depict  the  range  and  distribution  of  the  data 
and  the  presence  of  outliers. 

♦  Line  graphs  are  scatter  plots  with  lines 
connecting  the  data  points.  Line  charts 
are  appropriate  for  displaying  how  data 
changes  over  time.  Often,  the  dots  will  be 
connected  to  illustrate  this,  but  if  a  logical 
connection  does  not  exist,  the  dots  should 
not  be  connected.  To  avoid  scaling  effects, 
a  rectangular  plot  with  the  x-axis  about  1.5 
times  as  long  as  the  y-axis  is  appropriate. 

♦  Bar  charts  display  the  relationship 
between  categorical  variables  (x-axis) 
and  quantitative  variables  (y-axis).  For 
more  than  eight  categories,  use  a  rotated 
bar  chart.  Stacked  bar  graphs  should 

be  used  with  caution  as  it  is  difficult  to 
make  comparisons.  If  a  stacked  bar  graph 
is  needed,  the  category  that  requires 
comparisons  should  appear  on  the  bottom. 

♦  Probability  plots  help  to  determine  if  the  data 
follows  a  given  distribution.  If  the  data  forms 

a  somewhat  straight  line  on  the  plot,  then 
it  follows  a  normal  distribution.  Any  data 
that  does  not  appear  on  the  line  represents  a 
departure  from  normal  distribution. 

♦  Q;Q_  plots  are  a  type  of  probability  plot 
that  verify  if  two  similar  sets  of  data  can 
be  fit  with  the  same  distribution.  Q;Q_ 
plots  can  test  many  different  aspects  of 
the  data  and  can  also  assess  goodness  of 
fit  graphically  and  quantitatively  with  a 
probability  plot  correlation  coefficient. 

♦  Boxplots  are  good  for  depicting  the  median 
and  upper  and  lower  quartiles.  Some 
boxplots  also  depict  outliers.  Boxplots  can 
be  used  for  comparing  the  distribution  of 
the  data  of  two  or  more  groups  and  are 
especially  good  for  non-parametric  data 
since  means  are  not  appropriate  parameters 
for  non-normal  data. 

♦  Scatter  plots  are  one  of  the  most  efficient 
graphs  for  depicting  data  and  are  used  to 


detect  trends  or  correlations  between  two 
quantitative  variables.  The  x-axis  shows  the 
independent  variable  and  the  y-axis  shows 
the  dependent  variable.  Regression  lines 
quantitatively  describe  the  linear  relationship 
between  the  two  variables. 

Tables  usually  outperform  graphs  in 
reporting  small  data  sets  and  are  valuable 
for  reporting  exact  numerical  values.  It  is 
difficult  to  call  attention  to  a  series  of  data 
points  in  a  table  of  numbers;  graphing  the  data 
points  is  an  effective  way  to  highlight  them. 

Pie  charts  should  be  avoided  because  they 
do  not  allow  easy  comparisons  of  data  and 
make  it  difficult  to  discern  differences  in 
the  magnitude  of  each  slice.  They  also  use 
a  large  amount  of  ink  to  depict  a  relatively 
small  amount  of  data. 

In  General 

When  depicting  data  in  any  document,  use 
strict  rules  of  integrity  to  guard  MCOTEA’s 
reputation  as  an  independent  and  unbiased 
evaluator.  In  addition,  adhere  to  the  following 
guidelines: 

♦  Depictions  should  be  clear  and  concise. 

♦  Unnecessary  chart  decorations,  heavy  lines, 
overuse  of  color,  etc.  waste  space  in  the 
depiction  or  are  a  distraction.  All  ink  should 
be  used  efficiently  to  aid  in  conveying  what 
the  numbers  mean. 

♦  Avoid  the  use  of  3-D  charts,  which  add 
clutter  and  distort  the  data. 

♦  Clearly  define  what  the  numbers  represent 
on  the  graph. 

♦  Clearly  label  axes:  spell  out  acronyms  and 
abbreviations  on  the  labels. 

♦  Keep  gridlines  faint  or  delete  them  altogether. 

♦  Limit  numerical  labels  on  the  y-axis  to 
avoid  clutter.  Consider  labeling  each 
data  point  with  the  value  if  a  small  set  of 
numbers  is  depicted. 

♦  Show  error  bars  whenever  possible.  Use 
the  caption  or  the  graph  itself  to  inform  the 
audience  of  the  type  of  error  depicted. 
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♦  Legends  and  chart  titles  should  be  embedded 
into  the  chart  to  maximize  the  size  of  the 
area  used  for  displaying  the  data.  Legends  to 
the  side  of  a  graph  can  shrink  data  depiction 
space  and  should  be  avoided. 

♦  Avoid  cherry-picking  data.  Use  all  available 
data  if  possible.  Defend  the  reasoning  for 
not  using  all  data  within  the  text  of  the  report. 

♦  Axes  should  be  consistent  across 
comparisons  and  encompass  the  full  range 
of  the  data.  If  the  full  range  of  data  is  not 
depicted  on  the  axes,  explain  why. 

♦  Graphs  should  not  be  used  to  decorate  a  few 
numbers.  If  a  point  can  be  made  sufficiently 
with  words,  then  a  graph  is  not  needed. 

♦  Keep  design  variation  constant  to  maintain 
the  integrity  of  data  depiction  variation 
(e.g.,  don’t  vary  the  axes  intervals  or  the 
vertical  or  horizontal  scales). 

♦  Ensure  that  the  x-axis  is  about  1.5  times 
longer  than  the  y-axis  to  avoid  exaggerating 
the  data. 

♦  Do  not  include  a  title.  Figure  numbers  will 
be  added  beneath  the  graphic. 

♦  Ideally,  graphics  should  be  kept  in  their  own 
folder  and  submitted  separately  from  the 
text.  Ensure  that  graphics  are  numbered  for 
editorial  placement.  Ensure  that  all  graphics 
are  referenced  in  the  text. 


Klass  2008  and  Tufte  1996) 

MCOTEA  strives  for  consistency 
throughout  test  program  documentation. 
Formats  for  all  documents  are  similar  in 
terms  of  font  choice,  outlining  convention, 
table  formats,  and  basic  page  layout. 
Graphics  should  also  be  consistent  in 
terms  of  originating  software,  format, 
and  content.  Templates  for  the  most 
commonly  used  graphics  are  located  in 
each  document’s  template  folder. 

MCOTEA  Documents 

System  Evaluation  Plan 

The  SEP  contains  numerous  formulas 
as  well  as  standard  graphics  in  part  III, 
Evaluation  Methods. 

Formulas 

♦  Create  all  formulas  using  Equation  Maker 
in  Word  2007.  Use  lower  case  letters  for 
subscripts. 

♦  Set  the  font  to  Cambria  Math  italic  and  the 
font  size  to  10. 

♦  Use  the  one-line  table  (in  the  SEP 
template)  to  center  the  formula  on  the  page 
directly  beneath  the  text  that  leads  to  it. 

♦  Number  the  formula  in  the  table’s  right- 
hand  column  in  parentheses. 


Fig.  3-6-2. 

Mission  Capability 
Level  Standard  Graphic 
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Mission  Capability  Level  Value  Function 

The  MCL  is  used  to  evaluate  OE/OS/ 
OSur  for  all  systems.  The  graphics  that 
support  this  evaluation  are  standard  and  are 
available  in  the  SEP  template  folder.  For 
additional  information  regarding  the  MCL 
Value  Function,  see  chapter  3-1. 

Test  Plans 

Test  Plans  contain  a  number  of  standard 
tables  (as  found  in  the  template)  but  few 
unique  graphics  apart  from  the  tables. 
Graphics  that  do  appear  will  generally  be 
maps  or  Trial  Conduct  diagrams  unique  to 
each  program. 

♦  All  large  maps,  photos,  or  diagrams  should 
be  compressed  before  inserting  in  Word  to 
minimize  document  size.  All  images  should 
be  saved  in  .jpg  format. 

♦  .pdf  graphics  should  not  be  inserted  into 
Word.  If  a  .pdf  must  be  used,  the  graphic 
cannot  contain  typos  or  other  errors. 

♦  Diagrams  should  be  drawn  in  Visio,  if 
available,  or  Word.  In  either  case,  be  certain 
to  group  the  diagram  when  it  is  finished. 

To  group  a  diagram,  hold  Shift  while 
clicking  on  the  separate  parts  or  Select  All  if 
available.  When  all  parts  have  been  selected, 
right  click  and  select  “Group.” 

Reports 

Test  Reports  tend  not  to  contain  graphics 
other  than  tables.  However,  Evaluation 
Reports  do  contain  numerous  graphics  due 
to  data  analysis. 

When  formulas  are  used  in  reports,  follow 
the  SEP  guidelines.  Formulas  from  the 
SEP  can  be  copied  into  the  OER  to  save  time. 
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Sample  Depictions,  Good  and  Bad 


A  line  graph  (figure  3-6-3)  is  similar  to  a  scatter  plot;  however,  instead  of  using  a 
regression  line  to  show  the  relationship  between  the  variables,  the  points  are  connected 
by  a  line  to  show  how  the  data  changes  over  time.  The  x-axis  is  two  times  as  long  as  the 
y-axis  to  avoid  inadvertent  exaggeration  of  information. 

Figure  3-6-4  depicts  a  two-axis  column  line  chart  displaying  two  sets  of  data  using  three 
axes.  This  graph  is  easy  to  read  because  a  clearly  defined  legend  is  at  the  top  and  the  graph 
contains  no  distractions  from  the  data  being  presented. 


Series  1  (left  axis) 
■Series  2  (right  axis) 


Gray  out  lines 


Gray  out  borders  and 
■set  at  .25  points 


Jan  Feb  Mar  Apr  May  Jun  Jul  Aug  Sep  Oct  Nov  Dec 
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Fig.  3-6-4. 

Sample  Two-Axis 
Column  Line  Chart 
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Fig.  3-6-5. 
Sample  BarChart 


Fig.  3-6-6. 
Sample  Scatter  Plot 


Fig.  3-6-7. 
Sample  color 


Bar  charts  (figure  3-6-5)  are 
used  to  compare  categorical 
and  quantitative  data.  For 
example,  they  are  often  used 
to  compare  categories,  depict 
survey  results,  and  show  the 
distribution  of  data.  Error 
bars  should  be  included  in  bar 
charts. 


1  2  3  4  5  6 


Marine  Unit 


Figure  X.  Time  to  Process  a  Supply  Request  by  Marine  Unit 


In  a  scatter  plot  (figure  3-6-6), 
the  independent  variable 
appears  on  the  x-axis  and  the 
dependent  variable  appears  on 
the  y-axis.  A  regression  line 
often  appears  on  a  scatter  plot 
to  show  a  correlation  in  the 
data.  However,  the  data  must 
be  evaluated  to  determine  if  a 
correlation  is  intended. 


Figure  3-6-7  exemplifies  the 
use  of  color  when  color  is 
useful  in  depicting  data.  (Color 
by  itself  is  not  necessary  and 


may  in  fact  create  distraction.) 
When  choosing  colors  for  a 
depiction,  muted  colors  allow 
the  audience  to  focus  on  the 
data  rather  than  the  color 
scheme.  Also,  it  is  best  to 
refrain  from  using  red,  green, 
and  yellow,  as  these  colors 
create  a  stop  light  effect, 
which  is  not  appropriate  for 
evaluative  documents. 


Use  a  muted  color  so  as  not 
to  distract  from  the  data 


Average  Flow  Rate  -  4.05 


.<*  A  ^  g  ^  g  g 


Date/Time 


3-6-18 


Evaluate  Test  Results 


What  Not  to  Do 


experience  (yrs) 


l  experience  (yrs) 
1  Psuccess 


No  need  to  bold. 

(Does  not  aid  comprehension) 


These  graphs  are  not 
ideal  for  use  in  technical 
documents.  Three- 
dimensional  presentation 
does  not  clarify  data  and  in 
fact  can  obscure  important 
features  of  the  data. 


Fig.  3-6-8. 

Sample  3-D  Graph 


Clearly  define  what  the  numbers 
represent 


Pie  charts  do  not  allow 
easy  comparison  between 
pieces  of  data.  In  addition, 
too  many  design  elements, 
such  as  the  color  and  three- 
dimensional  presentation, 
interfere  with  interpretation. 


Fig.  3-6-9. 
Sample  Pie  chart 
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Fig.  3-6-10. 
Sample  three- 
dimensional  graph 


Fig.  3-6-11. 
Sample  graph 


Pictured  on  this  page 
are  two  graphs  depicting 
the  same  data  for  three 
Measures.  The  three- 
dimensional  graph  on 
top  captures  the  data 
but  creates  an  optical 
illusion  that  distracts  the 
viewer.  The  graph  below 
communicates  two  of 
the  three  Measures  on  a 
simple  line  chart;  it  has 
fewer  special  effects  than 
the  three-dimensional 
graph,  making  it  easier  to 
interpret.  However,  the 
two-dimensional  graph 
does  not  capture  the  third 
Measure,  Time  to  Advance. 


If  a  three-dimensional 
depiction  best  suits  the 
data,  the  picture  should 
be  generated  in  the  best- 
quality  graphing  program 
available.  If  a  high-quality 
program  is  not  available, 
the  data  is  better  left 
undepicted  than  shown  in 
either  of  the  two  examples 
on  this  page. 


Total  Target  Exposure  Time 
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Assessment  Process 


MCOTEA  Assessments 

MCOTEA’s  process  is  heavily  dependent 
on  performing  assessments  and  analyzing 
their  results  to  support  a  system’s 
overall  evaluation.  This  section  provides 
an  overview  of  the  different  types  of 
MCOTEA  assessments.  MCOTEA 
conducts  three  types  of  assessments: 
System,  Intermediate,  and  Operational, 
as  defined  in  the  following  sections. 
Assessments  occur  either  as  standalone 
events  (System  Assessment)  or  as  pre- 
IOT  events  (Operational  Assessment.) 
Common  to  all  assessments  are  the 
following  characteristics: 

♦  contractors  may  be  used  to  operate  and 
maintain  the  system 

♦  use  of  production-representative  articles  is 
not  required 

♦  technology  demonstrators,  prototypes, 
mock-ups,  engineering  development 
models,  or  simulations  may  be  used 

♦  OE/OS/OSur  is  not  determined 

The  results  of  any  assessment  are  sent 
to  the  PM  and  MDA  and  may  be 
distributed  further  at  the  discretion  of  the 
Director,  MCOTEA.  See  chapter  4  for 
reporting  requirements  and  deadlines. 

System  Assessments 

As  noted  in  the  introduction  to  this 
chapter,  System  Assessments  pertain  to 
programs  being  tested  or  examined  that 
do  not  require  operational  test,  such  as 
Quick  Reaction  Assessments  (QRA), 
Abbreviated  Acquisition  Programs 
(AAP),  ACAT  IV(M)  programs, 
and  other  non-Programs  of  Record. 
MCOTEA  uses  this  type  of  assessment 
to  answer  specific  questions  to  address 
risk  areas,  as  written  in  the  SAP. 


To  begin  the  System  Assessment  process, 
MCOTEA  writes  a  SAP,  which  serves 
as  a  framework  and  methodology  for 
performing  the  assessment  and  provides 
basis  for  eventual  analysis  of  assessment 
data.  After  performing  the  System 
Assessment,  MCOTEA  documents 
the  assessment  in  a  System  Assessment 
Report  (SAR). 

Table  3-1  provides  a  “menu”  of  possible 
ways  for  MCOTEA  to  be  involved  in 
System  Assessments,  along  with  the  prod¬ 
ucts  that  each  type  of  involvement  yields. 
Using  this  table,  MCOTEA  works  with 
the  program  sponsor  to  identify  the  exact 
nature  of  MCOTEA  involvement  in  the 
System  Assessment. 

Quick  Reaction  Assessment 

When  a  system  must  be  fielded  quickly 
an  Urgent  Operational  Need  Statement 
(UONS)  or  Urgent  Universal  Need 
Statement  (UUNS)  is  typically  issued  for 
the  system  in  development,  or  the  system 
may  be  granted  Rapid  Deployment 
Capability  (RDC)  status  by  ASN  (RDA). 
This  urgency  may  necessitate  modifying 
established  MCOTEA  OT&E  processes 
in  order  to  rapidly  procure  and  deliver  the 
urgently  needed  capability.  In  such  cases, 
the  program  sponsor  may  request  a  QRA 
from  the  Director,  MCOTEA.  The  QRA 
request  should  include  the  following: 

♦  purpose  of  the  System  Assessment  and 
the  specific  system  attributes  the  program 
sponsor  wants  assessed 

♦  time  available  for  the  System  Assessment 

♦  Concept  of  Employment 

♦  any  available  threat  documentation 

♦  resources  available  for  the  System  Assessment 

♦  forces  that  will  deploy  with  the  system 
before  IOC 
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Non-  Programs  of  Record,  AAPs,  ACAT  IV(M),  and  QRA 


Concur  with 
request  for  no 
OT;  no  further 
MCOTEA  program 
involvement 


MCOTEA  only 
observes  the 
testing  to  ensure 
a  quality  test  is 
executed 


MCOTEA-  led 
event  with  no 
assessment 


Provide 
assessment 
based  solely 
on  DT 


Provide 
assessment 
based  solely 
on  MCOTEA- 
led  test 


Provide 
assessment 
based  on 
both  DT  and 
a  MCOTEA- 
led  test 


ACAT  Designa¬ 
tion  Letter* 

SAP 


Observation  Plan 


Observation 

Report 

Test  Plan 


Test  Report 


SAR 


Operational  Task  Analysis  (OTA)  not  required 
x  x 


x 


x 


X 


X 


OTA  required 


x 


X 


X 


X 


X 


X 


X 


X 


X 


X 


X 


X 


X 


X 


X 


X 


X 


*  For  AAPs  and  ACAT  IV(M)  programs 


Table  3-1. 

MCOTEA 's  options 


for  involvement 


Execution  of  a  QRA  does  not  replace  the 
scheduled  operational  testing  as  approved 
in  the  TEMP  for  Programs  of  Record. 
Systems  in  RDC  status,  as  approved 
by  ASN  (RDA),  will  normally  undergo 
formal  OT&E  when  they  transition  to 


operational  requirements.  MCOTEA 
may  also  make  its  endorsement 
contingent  on  future  testing  to  ensure 
system  functionality  and  usability. 

Intermediate  Assessments 


with  System 
Assessments 


program  status. 

AAPs  and  ACAT  IV(M) 

By  definition,  AAPs  and  ACAT  IV(M) 
programs  (the  M  stands  for  “monitor”) 
do  not  require  operational  testing. 

They  do,  however,  require  a  MCOTEA 
endorsement  to  obtain  their  designation. 
As  part  of  the  designation  process, 
the  PM  requests  from  the  Director, 
MCOTEA,  a  written  endorsement  of  the 
proposed  acquisition  strategy.  (See  the 
appendix  to  this  section  for  details  of  the 
MCOTEA  endorsement  process.) 

AAPs  and  ACAT  IV(M)s  require 
adequate  DT  to  ensure  that  they  meet 
technical  goals  and  satisfy  the  user’s 


Intermediate  Assessments  pertain  to 
programs  at  the  ACAT  IV (T)  (Test) 
level  and  above.  They  are  governed  by  a 
SEP  and  are  most  commonly  performed 
after  DT  Observation.  Less  common  is 
an  Intermediate  Assessment  performed 
as  a  MCOTEA-led  DT  event.  Figure 
3-1  illustrates  the  iterative  process  of 
Intermediate  and  Operational  Assessments. 

DT  Observation 

MCOTEA  normally  observes  DT  events 
to  track  program  progress;  to  verify  that 
the  DT  event  was  executed  according 
to  plan;  and  to  verify  DT  data  results 
after  receiving  the  DT  report.  Properly 
performed  DT  Observation  enables 
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. 

V 


MCOTEA’s  Intermediate  and 
Operational  Assessment  Process 


IOT&E  Process 
“j  System  Evaluation  Plan 


2  Test  Concept,  TEMP 

Input,  and  FD/SC  Charter 
Development 


3  Test  Planning 


Assessment  Evaluation 

Assessment  Planning 

Reporting 

Assessment  Event 
)  Reporting 


Assessment  Event 


Repeat 

Assessment  Process  as 
Required 


OT  Execution 


6  System  Evaluation 
and  Reporting 


/r  Operational  Test 
Data  Reporting 


IOT&E  Process 


During  developmental  testing, 
system  components  are 
checked  to  ensure  that  they 
function  as  designed,  and  the 
system  is  checked  to  ensure 
that  it  meets  the  requirements 
derived  from  the  ICD/CDD/ 
CPD.  MCOTEA  generally 
uses  the  data  gathered  during 
DT  to  determine  if  the 
thresholds  in  the  approved 
capabilities  documentation 
have  been  demonstrated.  In 
addition,  aggregating  DT 
data  over  time  can  be  useful 
in  determining  a  system’s  OS 
and  OSur. 

As  with  any  assessment,  OE/ 
OS/OSur  is  not  determined. 
After  the  DT  Observation, 
MCOTEA  writes  an 
Observation  Report  and  later, 
after  receiving  the  DT  Report, 
an  IAR.  The  PM  and  MDA 
use  the  the  IAR  to  gauge  a 
program’s  progress  toward  IOT 
and  to  become  aware  of  any 
risks  to  program  success. 


Figure  3-1. 
Intermediate 
and  Operational 
Assessments 
Process 


MCOTEA  to  use  DT  data  in  overall 
system  evaluation  and  represents  an 
excellent  opportunity  for  MCOTEA  to 
collect  early  data  on  Suitability  issues. 

In  addition,  MCOTEA’s  participation 
gives  the  PM  insight  into  the  system’s 
developmental  progress,  materiel  maturity, 
and  readiness  to  enter  a  MCOTEA-led 
assessment  or  operational  testing  phase. 

DT  Observation  events  are  specified  in 
the  TEMP.  To  prepare  for  attending  the 
DT  event,  MCOTEA  prepares  a  DT 
Observation  Plan  for  internal  use,  based 
on  the  evaluation  questions  from  the  SEP 
pertinent  to  this  event.  MCOTEA  may 
also  participate  in  collaborative  planning 
of  the  DT  event,  but  only  DT  personnel 
execute  the  events  under  DT  observation. 


Operational 

Assessment 

MCOTEA  may  conduct  Operational 
Assessment  (OA)  to  demonstrate 
selected  system  performance,  with  user 
support  as  required.  An  OA  can  range 
from  a  “paper  assessment”  to  a  physical 
operational  test.  The  nature  of  the  OA 
is  described  in  the  TEMP.  An  OA 
can  be  conducted  at  any  time,  but  is 
normally  done  during  the  Engineering 
and  Manufacturing  Development  phase 
of  the  acquisition  cycle  to  evaluate 
selected  Issues,  KPPs,  and  other  system 
attributes.  An  OA  typically  focuses  on 
significant  trends  noted  in  developmental 
efforts,  programmatic  voids,  areas  of  risk, 
testability  of  capabilities,  and  the  ability 
of  the  program  to  support  adequate 
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operational  testing.  An  OA  does  not 
determine  OE,  OS,  or  OSur. 

Any  program  on  the  DOD  Oversight  List 
must  attain  acceptable  performance  in  an 
OA  before  entering  the  Production  and 
Deployment  phase  (DOD  2008).  An  OA 
provides  early  information  to  the  PM  and/ 
or  decision  maker  about  system  progress  in 
the  following  areas: 

♦  satisfying  capabilities  documentation 

♦  satisfaction  of  defined  Attributes  including 
KPPs  and  KSAs 

♦  readiness  for  LRIP 

♦  readiness  for  entry  into  IOT 

Characteristics  of  an  Operational 
Assessment  include  the  following: 

♦  May  also  be  used  to  support  program  reviews 
or  milestones 

♦  May  be  conducted  using  technology 


demonstrators,  prototypes,  mock-ups, 
engineering  development  models,  or 
simulations;  production-representative 
articles  not  required 

♦  May  use  typical  users  (Marines)  as  operators 

♦  May  be  conducted  under  actual  operational 
conditions 

♦  Does  not  substitute  for  IOT&E  needed  to 
support  full-rate  production  decisions 

Early  Operational  Assessment 

An  Early  Operational  Assessment  (EOA) 
is  similar  to  an  OA,  but  is  conducted  during 
the  Technology  Development  phase  of 
the  acquisition  cycle,  before  MS  B,  and 
is  typically  used  as  an  input  to  determine 
whether  a  system  should  continue 
development  and  proceed  to  Engineering 
and  Manufacturing  Development. 


Assessment  and  Test  Characteristics 


Characteristics 

Assessments 

Tests 

System 

Intermediate 

Operational 

IOT/  FOT/  MOT 

QRA/AAP  and 
ACAT  IV(M) 

DT  Observation 

EOA 

OA 

May  use  technology  dem¬ 
onstrations,  prototypes,  and 
mock-ups 

X 

X 

X 

X 

Production-representative 
models  requires 

X 

May  use  Marine  Operators 

X 

X 

X 

X 

Must  use  Marine  Operators 

X 

May  use  contractors  to  operate 
or  maintain  the  system 

X 

X 

X 

X 

May  be  conducted  under  actual 
operational  conditions 

X 

X 

X 

X 

Must  be  conducted  under 
actual  operational  conditions 

X 

Does  not  substitute  for  IOT 

X 

X 

X 

X 

Uses  representative  forces 
(both  friendly  and  opposing) 

X 

Employs  realistic  tactics  and 
targets  whenever  possible 

X 

Determines  OE/OS/OSUR 

X 
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Appendix  1.  T&E  Issues  Covered  During  Gate  Reviews 


Certain  Probability  of  Program  Success 
(PoPS)  items  associated  with  T&E 
will  be  covered  at  each  gate  review 
(Department  of  the  Navy  2008). 

Since  they  are  of  particular  interest  to 
MCOTEA,  they  are  shown  below. 

Gate  1 

Identified  alternatives  to  be  assessed  in 
the  AoA  can  be  evaluated. 

Gate  2 

♦  Key  stakeholders  have  been  identified  and 
have  agreed  to  participate  on  the  T&E 
WIPT 

♦  Plan  of  Action  with  Milestones 
(POA&M)  is  in  place  for  the  development 
of  the  Test  and  Evaluation  Strategy 

♦  Plan/schedule  to  accomplish  key  test 
activities  (prior  to  MS  B)  has  been 
developed  and  integrated  in  the  program 
master  schedule 

♦  Initial  review  of  test  resource  capabilities, 
including  ranges,  targets,  facilities, 
manpower,  Services,  Joint  assets,  and  other 
programs  indicates  that  resources  exist  and 
are  available  to  support  the  planned  T&E 
of  the  program 

♦  T&E  costs  have  been  indentified  and  are 
included  in  program  cost  estimates 

♦  All  KPPs,  KSAs,  and  other  Attributes  are 
measurable  and  testable 

♦  Preliminary  COIs  may  be  presented 

Gate  3 

♦  T&E  WIPT  has  been  formed 

♦  Test  and  Evaluation  Strategy  is  approved 
and  aligns  with  the  Acquisition  Strategy 
and  Systems  Engineering  Plan.  Critical 
comments  from  Navy/Marine  Corps 


staffing  have  been  adjudicated 

♦  Test  requirements  are  traceable  to 
capability  requirements  and  the  current 
threat  assessment 

♦  T&E  Strategy  includes  M&S  (as 
appropriate) 

♦  KPP,  KSA,  and  other  Attribute  threshold 
values  are  testable  and  measurable 

♦  Plan/schedule  to  accomplish  key  test 
activities  has  been  developed  and 
integrated  into  the  program  master 
schedule.  Adequate  calendar  time  exists 
based  on  historical  precedence 

♦  T&E  organizations  are  executing  key  test 
activities  on  or  ahead  of  schedule 

♦  Review  of  test  resource  capabilities, 
including  ranges,  targets,  facilities, 
manpower,  Services,  Joint  assets,  and 
other  programs  has  been  conducted. 

Gaps  have  been  identified  (if  any)  and 
mitigation  plans  have  been  established 

♦  T&E  costs  have  been  indentified  and  are 
included  in  program  cost  estimates 

♦  Initial  COIs  may  be  presented 

Gate  4 

♦  TEMP  is  approved  and  aligns  with 
the  Acquisition  Strategy  and  Systems 
Engineering  Plan.  Critical  comments 
from  Navy/Marine  Corps  staffing  have 
been  adjudicated 

♦  Test  requirements  are  traceable  to 
capability  requirements  and  the  current 
threat  assessment 

♦  TEMP  identifies  M&S  requirements  and 
utilization 

♦  KPP,  KSA,  and  other  Attribute  threshold 
values  are  testable  and  measurable 

♦  T&E  organizations  are  executing  key  test 
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activities  on  or  ahead  of  schedule 

♦  T&E  costs  have  been  indentified  and  are 
included  in  program  cost  estimates 

♦  Review  of  test  resource  capabilities, 
including  ranges,  targets,  facilities, 
manpower,  Services,  Joint  assets,  and  other 
programs  has  been  conducted.  Gaps  have 
been  identified  (if  any)  and  mitigation  plans 
have  been  established 

♦  EOA/preliminary  test  results  have  not 
identified  any  significant  performance  risks/ 
issues 

♦  T&E  requirements  for  the  RFP  have  been 
finalized 

♦  Deficiency  identification  and  tracking 
system  has  been  identified  for  the  program 

♦  Final  COIs  may  be  presented 

Gate  5 

♦  TEMP  is  approved,  current,  and  aligns 
with  the  Acquisition  Strategy  and  Systems 
Engineering  Plan 

♦  Test  requirements  are  traceable  to  capability 
requirements  and  the  current  threat 
assessment 

♦  TEMP  identifies  Modeling  and  Simulation 
requirements  and  utilization 

♦  KPP,  KSA,  and  other  Attribute  threshold 
values  are  testable  and  measurable 

♦  T&E  organizations  are  executing  key  test 
activities  on  or  ahead  of  schedule 

♦  Test  resource  capabilities,  including  ranges, 
targets,  facilities,  manpower,  Services, 

Joint  assets,  and  other  programs  have  been 
assessed  and  can  support  planned  test 
activities 

♦  T&E  costs  have  been  identified  and  are 
included  in  program  cost  estimates 

♦  Early  Operational  Assessment/preliminary 
test  results  have  not  identified  any 
significant  performance  risks/issues 


♦  The  RFP  contains  T&E  requirements, 
including  government  review  and  oversight 
and  provisions  for  the  Integrated  Test  Team, 
as  appropriate 

♦  Deficiency  identification  and  tracking  has 
been  developed  and  is  being  used 

Gate  6 

♦  TEMP  is  approved,  current,  and  aligns 
with  the  Acquisition  Strategy  and  Systems 
Engineering  Plan 

♦  Test  requirements  are  traceable  to  capability 
requirements  and  the  current  threat 
assessment 

♦  TEMP  identifies  M&S  requirements  and 
utilization 

♦  KPP,  KSA,  and  other  Attribute  threshold 
values  are  testable  and  measurable 

♦  T&E  organizations  are  executing  key  test 
activities  on  or  ahead  of  schedule 

♦  Test  resource  capabilities,  including  ranges, 
targets,  facilities,  manpower,  Services, 

Joint  assets,  and  other  programs  have  been 
assessed  and  can  support  planned  test 
activities 

♦  T&E  Costs  have  been  identified  and  are 
included  in  program  cost  estimates 

♦  Deficiency  identification  and  tracking 
accurately  displays  the  current  status  on  the 
resolution  of  deficiencies  identified  during 
testing  prior  to  IOT&E 

♦  Major  Deficiencies  and  OTA 
recommendations  identified  in  IOT&E 
and  FOT&E  reports  are  available  for 
review.  This  includes  the  approval  of  the 
dispensation  of  those  deficiencies  that  the 
program  recommends  taking  no  action  to 
correct,  or  reassigned  to  another  developing 
activity  due  to  System  of  Systems  interfaces 
and  compatibility. 
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Appendix  2.  Abbreviated  Acquisition  Program/ 
AC  AT  IV(M)  Endorsements 


AAP/ACAT  IV(M)  programs  enter 
MCOTEA  through  the  S-2  for  tracking 
purposes  only  and  are  immediately 
forwarded  to  the  appropriate  Division  for 
action. 

The  Division  evaluates  the  request  using 
the  guidance  and  document  checklist 
in  this  appendix.  After  completing  the 
evaluation,  the  Division  generates  and 
staffs  an  endorsement  letter  for  the 
Director’s  signature,  using  the  samples  at 
the  end  of  this  appendix. 

The  Division  returns  the  final  package 
to  the  S-l  for  appropriate  formatting, 
document  control  numbering,  routing,  and 
distribution  of  final  correspondence.  The 
S-l  provides  copies  to  the  Division,  the 
S-2,  and  requesting  activity. 

AAP/ACAT  IV(M)  Review  Process 
and  Endorsement  Considerations 

Program  Research 

♦  Ensure  that  supporting  documentation  is 
provided  and  reviewed 


♦  Identify  the  system  and  how  will  it  be  used 
(Mission,  KPPs,  high  visibility  thresholds) 

♦  Identify  the  Program  Office’s  risk 
mitigation  strategy 

Justification  for  Endorsement 

Base  the  justification  on  the  following: 

♦  Review  of  test  plans  and/or  test  reports 

♦  Observation  of  testing  and/or  user 
evaluations 

Include  the  justification  in  the 
Endorsement. 

Endorsement  may  be  contingent  on 
the  following: 

♦  Issues  identified  during  review  are  addressed 

♦  Future  testing  occurs  to  ensure  system 
functionality  and  usability 

-Developmental  Test 

-User  Evaluations 

-DT  Observation 

-MCOTEA-led  Testing 


AAP  and  ACAT IV(M)  Documentation  Checklist 


The  Commander,  MCSC  ACAT  IV(M)  or  AAP 

Request  to  Director  MCOTEA  should  contain 

the  following  information: 

♦  Purpose  (AAP  designation  and  concurrence) 

♦  Brief  description  of  the  program 

♦  Summary  of  projected  program  life  cycle  costs. 
Does  not  have  to  be  an  “independent”  Life 
Cycle  Cost  Estimate,  but  needs  to  cover  total 
ownership  cost  of  the  program 

♦  A  cost  and  funding  summary:  estimated 
cost  versus  budget  figures.  Use  the  Director, 
Financial  Management  budget  figures,  since 
that  office  must  concur  with  the  AAP  request 


♦  A  schedule  or  outline  of  significant 
program  events  that  includes  objectives  and 
thresholds  as  appropriate  if  contained  in  the 
requirements  document 

♦  A  discussion  of  the  developmental  testing 
(if  any)  planned  for  the  program.  Discuss 
all  testing  MCOTEA  plans  to  conduct 
and  present  the  results  of  any  testing 
already  conducted  by  the  contractor  and/or 
government,  including  any  user  events.  Test 
data  may  be  very  important  to  MCOTEA’s 
concurrence  with  the  AAP  request.  Though 
not  mandatory,  a  requirements  test  matrix  is 
beneficial  when  presenting  upstream 
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UNITED  STATES  MARINE  CORPS 
MARINE  CORPS  OPERATIONAL  TEST  AND  EVALUATION  ACTIVITY 
2032  BARNETT  AVENUE 
QUANTICO,  VIRGINIA  221 34-501 4 


3980/ 1 0/MCOTEA 
Date 


From:  Director,  Marine  Corps  Operational  Test  and  Evaluation  Activity 
To:  Commander,  Marine  Corps  Systems  Command 

Subj :  ACQUISITION  PROGRAM  CATEGORY  IV(M)  OR  ABBREVIATED  ACQUISITION 
PROGRAM  (AAP)  CONCURRENCE/NONCONCURRENCE  FOR  THE  [PROGRAM 
NAME] 

Ref:  (a)  SECNAVINST  5000.2D 

(b)  MCSC  ltr  xx  Ser  of  date  (request  letter) 

(c)  Others  as  appropriate 

Enel:  (1)  As  appropriate 

1 .  In  accordance  with  reference  (a),  this  concurrence  letter  addresses  the  proposed  ACAT  IV(M) 
or  AAP  outlined  in  reference  (b).  As  requested,  MCOTEA  has  reviewed  the  [Program  Name]. 
References  (c)  and  others  identify  the  required  capabilities  of  the  program.  [Note-these 
references  may  already  be  part  of  the  request  message;  if  so,  statement  would  be  references  x  and 
y  of  reference  (b)  identify  the  required  capabilities], 

2.  MCOTEA  concurs/does  not  concur  with  the  designation  of  the  [program  name]  as  an  ACAT 
IV(M)  or  AAP  that  does  not  require  independent  operational  testing.  MCOTEA  bases  it  decision 
on  the  information  presented  in  references  (b)  and  (c).  MCOTEA’s  concurrence  is  contingent 
upon  its  membership  in  the  Test  and  Evaluation  Working-level  Integrated  Product  Team  (T&E 
WIPT)  and  concurrence  of  the  program  name  Test  and  Evaluation  Master  Plan  (TEMP)  or  the 
[program  name]  Test  and  Evaluation  Strategy  (TES).  [Note-if  MCOTEA  has  no  contingencies 
then  drop  the  requirement  for  being  on  the  T&E  WIPT  and  concurrence  of  the  TEMP  or  TES] 

3.  Should  the  functionality  or  scope  of  this  program  change,  MCOTEA  should  be  re-engaged  to 
evaluate  if  its  concurrence  with  an  ACAT  IV(M)  or  AAP  designation  is  still  appropriate  and  to 
support  the  program  name  in  any  test  strategy  or  risk  mitigation  efforts. 

4.  The  MCOTEA  point  of  contact  is  [NAME]  at  (703)  xxx-xxxx  or  name@usmc.mil. 


[DIRECTOR] 


Copy  to: 

MCSC  (PG  requesting,  PM  requesting,  ACPROG,  DC  SIAT) 
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♦  A  reference  to,  or  a  copy  of,  the  DC, 
CDSd-validated  requirement  for  the 
program.  For  new-start  AAPs  and  IT 
AAPs,  the  requirement  may  take  the  form 
of  a  Statement  of  Need  or  Capability 
Document  such  as  an  ICD,  CDD,  CPD, 
which  outlines  the  requirement 

♦  Rationale  for  recommending  AAP 
designation 

Desired  Supporting  Documentation 

Test  concepts/plans,  government  or  contractor 
(draft/final  as  available).  (If  MCOTEA  concurs 
with  IV(M)  decision,  MCOTEA  will  provide 
input/recommendations  to  MCSC  during 
development.) 


DC,  CD&I  COE  or  OMS/MP  (may  be 
satisfied  with  COE/CONOPS  contained  in 
requirements  document  if  sufficient  detail  is 
provided) 

Test  reports  and/or  evaluations,  government  or 
contractor  (draft/final  as  available) 

MCOTEA  Output 

Rough  Order  of  Magnitude  (Cost  Estimate, 
Issues/Concerns)  (if  applicable) 

MCOTEA  Concurrence/Non-Concurrence 
Letter 

Lessons  Learned 


References 

Department  of  the  Navy.  2008.  Naval  PoPS  Crite¬ 
ria  Handbook,  Version  1.0. 
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Documentation 


MCOTEA  produces  a  variety  of 
documentation  throughout  the  life  of  a 
program.  Each  step  of  MCOTEA’s  test  and 
evaluation  process  generates  at  least  one 
document  product,  ranging  from  relatively 
simple  letters  to  comprehensive  test  plans 
and  evaluation  reports.  This  chapter  explains 
the  nature  of  each  document,  its  schedule, 
author,  and  content. 

MCOTEA’s  Standardized 
Approach  to  Documentation 

MCOTEA  produces  nearly  all  of  its 
test  and  evaluation  documentation  in 
a  repeating,  standardized  format  that 
supports  scientific  and  technical  reporting. 
Repeating  the  format  allows  each 
document  to  “feed”  the  next  one,  creating 
ease  of  use,  consistency,  and  traceability 
throughout  a  program’s  T&E  history. 
MCOTEA  T&E  documents,  in  typical 
order  of  creation,  are  as  follows: 

Early  Test  and  Evaluation  Planning 
Documents  for  Operational  Test 

♦  System  Evaluation  Plan  (SEP).  Pre-MS  B; 
sets  forth  the  evaluation  plan  the  program 
will  follow  for  the  duration;  prepared  by  OA 
and  test  team. 

♦  Test  Concept.  Working  document  prepared 
in  PowerPoint  for  briefing  purposes, 
developed  by  the  OA,  statistician,  and  test 
team  after  the  SEP  and  before  the  TEMP. 

♦  FD/SC  Charter.  Written  by  the  OA  and  test 
team  with  MCSC  and  DC,  CD&J. 

♦  Request  for  Clarification  Letter.  Written 
by  the  test  team,  addressed  to  DC,  CD&I 
when  portions  of  a  capabilities  document 
are  unclear. 

♦  Feasibility  of  Support.  Naval  message 
outlining  requirements  for  test  personnel 
and  facilities;  generated  by  OTPO/S-3. 

Plans  and Repoits  Pertaining  to  Developmental  Test 

♦  DT  Observation  Plan.  Written  by  the  test 
team,  MCOTEA’s  plan  for  observing  DT 
events. 


♦  Observation  Report.  Written  by  the  test 
team,  documents  the  adequacy  of  DT 
execution  with  no  judgment  or  conclusion; 
usually  written  before  test  results  are 
available;  copy  sent  to  PM. 

♦  Intermediate  Assessment  Report.  Based 
on  OTPO/TM  concurrence  with  DT 
Report;  OTPO/TM/OA  prepare;  addressed 
to  the  PM  and  the  MDA. 

Plans  Pertaining  to  Operational  Test  Events 

These  plans  require  a  SEP  as  their  basis. 

♦  Early  Operational  Assessment  Test  Plan 
(EOATP).  Specifies  test  logistics  and  the 
detailed  planning  of  test  trials  at  the  Issue/ 
Subtask  level. 

♦  Operational  Assessment  Test  Plan  (OATP). 

Specifies  test  logistics  and  the  detailed 
planning  of  test  trials  at  the  Issue/Task  level. 

♦  Initial  Operational  Test  Plan  (IOTP). 

Specifies  test  logistics  and  the  detailed 
planning  of  test  trials  at  the  COI/Mission 
level  and  lower  levels  as  required. 

♦  Follow-on  Operational  Test  Plan  (FOTP). 

Specifies  test  logistics  and  the  detailed 
planning  of  test  trials  of  any  post-IOT 
events. 

♦  Multi-Service  Operational  Test  Plan 
(MOTP).  Specifies  test  logistics  and  detailed 
planning  for  MCOTEA’s  participation  in 
Multi- Service  testing. 

Reports  Pertaining  to  Operational  Test 

♦  Test  Data  Report.  Packages  the  test  data 
from  the  event  (before  analysis). 

♦  Operational  Test  Agency  Assessment 
Report  (OAR).  Evaluation  report  that 
follows  an  EOA/OA;  stops  short  of  OE/ 
OS/OSur;  does  not  support  a  milestone 
decision;  OTPO/TM/OA  prepare. 

♦  Operational  Test  Agency  Evaluation 
Report  (OER).  Documents  final  system 
evaluation  after  IOT;  provides  OE/OS/ 
OSur  designation;  OTPO/TM/OA 
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prepare;  addressed  to  ACMC. Operational 
Test  Agency  Follow-On  Evaluation  Report 
(OFER).  Used  after  FOT&E  as  the  final 
evaluation  report;  OTPO/TM/OA  prepare. 

♦  Operational  Test  Agency  Milestone 
Assessment  Report  (OMAR).  Evaluation 
report  for  EOA/OA  that  supports  a 
milestone  decision;  OTPO/TM/OA  prepare. 

Documents  Pertaining  to  Non-Typical  Programs 

♦  System  Assessment  Plans,  System 
Assessment  Test  Plans,  System 
Assessment  Test  Reports,  and  System 
Assessment  Reports.  All  documents  follow 
the  regular  templates,  tailored  for  less  detail. 
See  sample  templates  later  in  this  chapter. 

Plans  and  Reports  Pertaming  to  Joint  or 

Multi-Service  Test  Events 

♦  Documents  (and  schedules)  conform  to 
those  of  the  lead  Service.  If  MCOTEA  is 
the  lead,  documents  continue  to  conform  to 
standard  plan  and  report  templates,  with  the 
addition  of  annexes  for  Joint  contributions. 

Plans  and  Reports  Pertaining  to  the  V&V 

Process.  (Refer  to  MIL-STD-3022.) 

♦  Accreditation  Plan 

♦  Accreditation  Report 

♦  Accreditation  Decision  Letter 


Document  Approval 
Process 

All  documents  proceed  through 
MCOTEA’s  chain  of  command  for 
approval,  and  mostT&E-related  documents 
require  the  Director’s  signature  (see  table 
4-1,  next  page).  All  program  documentation 
must  be  edited  before  entering  the  approval 
process.  While  constructing  a  program’s 
POA&M,  the  OTPO/TM  must  include 
time  for  the  CRB  to  receive  and  review 
documents. 

The  lead  time  for  submitting  documents 
to  the  CRB  can  flex  depending  on  the 
program,  to  be  determined  during  test 
planning.  The  CRB  is  composed  of  the 
Scientific  Advisor,  the  Chief  of  Test, 
and  the  S-2  Lead.  The  board  reviews  the 
draft  document  for  technical  content  and 
adherence  to  MCOTEA  process,  format, 
and  standards.  After  CRB  approval  and  any 
required  changes,  the  document  is  ready 
for  the  Director’s  review.  Note  that  the 
timelines  in  table  4-1  include  signature  time. 


Base  Templates 

The  base  template forDT  Observation  Plans  contains  the 
following  sections: 

1.  Purpose 

2.  Background  (problem  definition  and  system 
description) 

3.  Schedule 

4.  Organization 

5.  Evaluation  Questions  (COIs,  Issues) 

Annexes  as  specified 

The  base  template  for  posttest  reports  contains  the following 
paragraphs: 

1.  Purpose 

2.  Background  (problem  definition  and  system 
description) 

3.  Scope 

4.  Objectives 

5.  Assumptions 

6.  Limitations 

7.  Methods 


8.  Results 

9.  Insights 

10.  Conclusions 

11.  Recommendations 

12.  References 
Annexes  as  specified 

The  following  documents  do  not follow  a  base  template 
because  they  are  not  produced  solely  by  MCOTEA  or  they 
follow  mandated  content: 

♦  TEMP  (see  Defense  Acquisition  Guidebook  for  outline). 

♦  FD/SC  Charter  (samples  are  in  Templates  folder) 

♦  Accreditation  Plan 

♦  Accreditation  Report 

Four  documents  are  based  on  unique  formats  that 
support  each  document’s  purpose: 

♦  SEP/SAP 

♦  Test  Concept  (PowerPoint  brief) 

♦  Clarification  Letter  (follows  correspondence  format) 

♦  Accreditation  Decision  Letter 
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Table  4-1. 
Standard 
Documentation 
Schedule 


Task  Responsible  Schedule/  CRB  Approval  Additional 

Entity  Planning  Factor  Required  (Y/N)  Approval  Steps/ 

(Timeline  includes  signature)  Process 

System  Evaluation  Plan/ 
System  Assessment  Plan 

OA 

Program  entry  +  up  to  120 
calendar  days  (80  working  days). 
Time  limit  is  for  ACAT 1  only;  all 
others  will  be  produced  in  less 
time. 

Y 

Director  signs. 

Observation  Plan 

OTPO/TM 

From  receipt  of  final  DT  Plan  +  5 
working  days. 

N 

Div  Head  signs. 

Observation  Report 

OTPO/TM 

14  calendar  days  upon  return  from 
event  (10  working  days). 

N 

Div  Head  signs  and 
sends  copy  to  PM. 

Elapsed  time  without 
receipt  of  Developmental 
Test  Report/generate 
letter  to  MCSC  requesting 
report 

OTPO/TM 

Based  on  expected  due  date  for 
receipt  of  report  as  stated  in  TEMP 
or  30  days  after  test  completion  if 
not  specified;  follow  up  within  5 
days  of  report  being  late. 

N 

Div  head  signs  and 
sends  to  PM. 

Intermediate  Assessment 
Report  (IAR)/System 
Assessment  Report  (SAR) 

OA 

14  calendar  days  (10  working 
days)  after  receipt  of  test  report; 
reports  can  be  distributed 
individually  or  aggregated  after 
last  required  event,  depending  on 
program.  Reports  may  be  timed 
for  use  at  Gate  Reviews. 

Y 

Director  signs;  sent  to 
PMandMDA. 

Feasibility  of  Support 
Message 

0TP0/S-3 

NLT  6  months  before  test.  This  is  a 
standard  naval  message. 

N 

Test  Concept  (internal 
planning  brief) 

OTPO/TM 

Due  with  MC0TEATEMP 
submissions. 

N  (Yes  if  D0T&E 
Oversight) 

Clarification  Letter 

OTPO/TM 

Sent  after  MC0TEA  reviews 
CDD/CPD;  may  need  to  send 
multiple  letters  if  newer  versions 
of  capabilities  documents  are 
released  or  to  obtain  concurrence 
on  new  standards  associated  with 
Issues. 

N 

Director  signs. 

Test  and  Evaluation 

Master  Plan  (including 

Test  Concept) 

OTPO/TM 

As  designated  by  T&E  WIPT,  up  to 

40  working  days. 

Y 

Director  signs  for 

MC0TEA. 

FD/SC  Charter 

OTPO/TM 

Must  be  available  14  calendar  days 
(10  working  days)  before  first  test 
event  identified  to  collect  RAM 
data. 

Y 

Signature  at  appropriate 
level.  Director  or  Div 

Head  signs. 

Operational  Test 

Readiness  Board  (OTRB) 

OTPO/TM 

Schedule  90  calendar  days  (60 
working  days)  before  training 
begins. 

N 

Director  concurs  or  does 
not  concur  with  brief. 

4-4 


Documentation 


Task 

Responsible 

Entity 

Schedule/ 

Planning  Factor 
(Timeline  includes  signature) 

CRB  Approval 
Required  (Y/N) 

Additional 
Approval  Steps/ 
Process 

Test  Plan 

OTPO/TM 

Must  be  available  14  calendar 
days  before  OTRR  (10  working 
days)  (for  I0T  and  OA)  or  before 
test  (for  AAP,  ,ACAT  (IV)H,  and 

QRA. 

Y 

Director  signs  for 

MCOTEA. 

OTRR  Brief 

OTPO/TM 

30  calendar  days  before  training 
in  support  of  operational  test. 

N 

Director  concurs  or  does 
not  concur  with  brief. 

In-Process  Review 

Meeting 

OTPO/TM 

Schedule  NLT 14  calendar  days 
after  all  test  data  has  been 
collected.  Two  working  days. 

N/A 

Test  Data  Report 

OTPO/TM 

9  calendar  days  (7  working  days) 
after  test  completion. 

Y 

Director  signs.  Sent  to 
D0T&E  for  oversight 
programs. 

Released  to  others  at 
Director's  discretion. 

0AR/0ER/0FER 

OA 

MCOTEA  preference  is  45  calendar 
days,  30  working  days  after  test, 
including  signature. 

Y 

OER  and  0FER  addressed 
to  ACMC.  Director  signs. 
Copy  to  PMandMDA. 
Copy  to  D0T&E  for 
oversight  programs. 

Director  signs  OAR.  Sent 
to  PMandMDA. Copy 
to  D0T&E  for  oversight 
programs. 

Lessons  Learned 

OTPO/TM 

Complete  NLT  45  calendar  days 
following  OER  signature  (20 
working  days). 

N 

Archiving 

0TP0/S-1 

NLT  30  calendar  days  following 
MCOTEA  Program  Closure. 

N 

Accreditation  Plan  (VV&A 
Process) 

ACA 

30  calendar  days  before  V&V 
activity  (assuming  MCOTEA  early 
involvement). 

Y 

COT  signs. 

Accreditation  Report 

ACA 

30  calendar  days  before  OTRB. 

Y 

COT  signs. 

Accreditation  Decision 
Letter 

ACA 

30  calendar  days  before  OTRB. 

Y 

Director  signs. 

Note:  if  MCOTEA  performs  V&V  (unusual  but  possible),  the  ACA  is  also  responsible  for  a  V&V  Plan  and  V&V  Report.  Both  require  CRB  and 
the  COT  signs. 
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General  Guidance  for  Writing  MCOTEA  Documents 
Templates 

Templates  and  samples  for  documentation  discussed  in  this  chapter  are  contained  in  the 
Templates  folder  stored  on  the  MCOTEA  shared  drive.  The  templates  are  saved  as  Read 
Only  to  preclude  overwriting  the  master  version. 

Cover  Pages 

The  covers  of  all  MCOTEA  documents  contain  a  Distribution  Statement,  based  on  the 
document's  content  and  purpose.  Distribution  statements  are  used  in  lieu  of  “For  Official 
Use  Only.”  A  complete  explanation  of  cover  markings  is  contained  in  Annex  A  of  this 
chapter. 

Executive  Summaries 

Most  MCOTEA  documents  are  short  enough  that  an  Executive  Summary  is  not  required. 
However,  a  summary  should  be  included  when  the  main  body  of  a  document  exceeds  four 
pages.  Executive  Summaries  are  usually  included  with  an  OER  as  well,  regardless  of  length, 
since  this  document  is  sent  to  the  Assistant  Commandant. 

The  following  paragraph  headers  are  used  for  an  Executive  Summary: 

1.  Purpose 

2.  Background 

3.  Scope 

4.  Conclusions 

5.  Recommendations  (include  top  three  only) 

The  summary  must  not  exceed  one  page  in  length  and  should  not  carry  any  information 
or  ideas  that  are  not  contained  in  the  main  document  itself.  The  best  way  to  write  an 
Executive  Summary  is  to  finish  the  main  document  first,  then  copy  and  paste  key  ideas 
from  the  paragraphs  with  the  same  headers  noted  above  into  the  summary. 

Graphics 

Guidance  for  creating  original  graphics  to  be  used  in  MCOTEA  documents  is  contained 
at  the  end  of  chapter  3-6.  Graphics  coming  from  other  sources  should  be  large  enough 
(generally  1  MB  or  more)  to  reproduce  well. 

Annexes 

The  template  for  each  document  lists  any  required  annexes.  To  support  consistency  among 
MCOTEA  documents  and  to  keep  them  as  streamlined  as  possible,  no  additional  annexes 
should  be  included  without  CRB  concurrence. 

Editorial  References 

MCOTEA  abides  by  a  number  of  standard  editorial  references,  such  as  the  Navy 
Correspondence  Manual  (for  letters  and  memos),  the  Government  Printing  Office  Style 
Manual  for  general  guidance,  and  the  Chicago  Manual  of  Style  for  all  else.  MCO  5216.20 
provides  additional  Marine  Corps-specific  guidance  on  style  and  usage. 
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Guidance  for  Writing  Specific  Documents 

Hie  following  pages  provide  a  detailed  look  at  templates  for  MCOTEA  documents. 
Margin  notes  provide  additional  suggestions  and  commentary. 


System  Evaluation  Plan/ 
System  Assessment  Plan 

Author:  OA,  with  test  team  assistance 

The  SEP  (or  SAP  for  assessments)  sets 
forth  the  evaluation  plan  that  a  program 
will  follow  from  its  inception  to  the  final 
report.  Chapter  3-1  contains  detailed 
information  about  creating  the  SEP’s 
content. 

The  SEP  is  organized  into  three  primary 
sections:  I.  System  Definition;  II. 
Evaluation  Framework;  and  III.  Evaluation 
Methods.  Typically,  the  OTPO/Test 
Manager  write  section  I  of  the  SEP  and 
coordinate  with  the  OA  on  sections  II  and 
III.  These  sections  are  indented  within  a 
base  template  beginning  with  Purpose. 

Section  I,  System  Definition,  is  generally 


1-3  pages  in  length  and  helps  prepare 
the  analyst  to  ask  the  right  questions  for 
developing  the  evaluation  methods. 

Section  II,  Evaluation  Framework, 
contains  the  COIs,  lower  level  Issues, 
their  Measures,  and  then  lists  Issues  and 
Screening  Criteria  in  table  1. 

Section  III,  Evaluation  Methods,  contains 
two  parts,  Analytic  Model  and  Decision 
Model.  The  Analytic  Model  focuses  on 
COIs  set  forth  in  the  framework,  and 
the  Decision  Model  speaks  to  the  MCL. 
Section  III  is  the  mathematical  portion  of 
the  SEP. 

Before  beginning  to  write  a  SEP,  the  test 
team  will  find  it  useful  to  know  where  to 
place  the  Measures  they  will  develop  for 
the  system  under  test.  Table  4-2  outlines 
the  possible  placement  of  Measures. 


Table  4-2.  Measure  Placement  in  the  SEP 


Measure 

Type 

Measure  Use 

Evaluation 

Framework 

Issues  & 
Screening 
Criteria  Table 

Analytic 

Model 

MOE 

• 

*  Only  if 

Measure  hasCDD 
or  CPD  threshold 

*  If  used  in 
determining 
OE 

MOP,  MOS, 
MOSur 

HasaCDD  or  CPD  threshold 

• 

Measurement  will  be  taken  in 

OT  and  is  not  an  MOE.  Includes 
Measures  used  to  diagnose  OT  results, 
e.g.,  surveys,  performance,  suitability 

• 

• 

Measurement  will  be  taken  in  DT  and/ 
or  EOA,  OA,  or  other  MCOTEA-led  test 
other  than  I0T,  MOT,  or  FOT 

• 

Used  to  construct  Analytic  Model 

• 

• 

4-7 


Chapter  4 


In  section  I,  reference  the  source  of 
statements  made  about  the  system 
and  do  not  copy  “sales  talk”  from 
manufacturers’ websites.  For  example, 
saying  that  the  system  “will  greatly 
enhance”  situational  awareness  or 
that  the  system  “will  provide  highly 
efficient  capabilities”  presupposes 
the  outcome  of  testing.  Appropriate 
language  is  “the  system  is  intended  to 
enhance”  or  “is  designed  to  improve 
efficiency.”  Remain  neutral  by  not 
using  adverbs. 


System  Evaluation/Assessment  Plan 
[System  Name] 

1.  Purpose.  This  System  Evaluation  Plan  (SEP)  provides  evaluation, 
execution,  and  management  guidance  for  the  [system].  Within  this  plan  are 

a  System  Definition,  an  Evaluation  Framework,  and  the  Evaluation  Methods 
for  [include  all  that  apply:  MCOTEA-led  Intermediate  Assessments, 
Operational  Assessments,  and/or  Operational  Evaluations  of  the  [system]]. 

System  Assessment  Plan  Purpose:  This  System  Assessment  Plan  (SAP)  provides 
execution  and  management  guidance  for  assessing  the  [system].  Within  this  plan 
are  a  System  Background,  Assessment  Framework,  and  Assessment  Methods. 

2.  Scope.  This  SEP  covers  the  breadth  of  evaluation  questions  that  must 
be  answered  over  time  to  conclude  OE,  OS,  and  OSur.  MCOTEA  will  use 
data  obtained  from  [include  all  that  apply:  developmental  tests,  MCOTEA- 
led  tests,  and  operational  tests]  in  the  preparation  of  [include  all  that  apply: 
Intermediate  Assessment  Reports  (IAR),  an  Operational  Test  Agency 
Assessment  Report  (OAR),  an  Operational  Test  Agency  Evaluation  Report 
(OER). 

System  Assessment  Plan  Scope:  This  Assessment  Plan  covers  specific  evaluation 
questions  as  outlined  in  the  Assessment  Framework.  MCOTEA  will  use  data 
obtained from  [include  all  that  apply:  developmental  tests  and/or  MCOTEA-led 
tests]  in  the  preparation  of  a  System  Assessment  Report  ( SAR) ]. 

I.  System  Definition.  [This  section  begins  with  a  definition  of  the 
capabilities  gap  that  the  materiel  solution  is  meant  to  address.  The  section 
should  conclude  with  a  description  of  the  system  being  evaluated.  For 
purposes  of  the  SEP,  a  system  is  defined  as  the  Marine  unit  or  crew  and 
their  equipment,  which  includes  the  materiel  solution  that  will  be  used  to 
accomplish  missions.  In  this  section,  the  author  accounts  for  four  main  ideas: 
the  system  and  its  purpose,  the  system’s  position  in  the  hierarchy  of  systems, 
the  system’s  limits  or  boundaries,  and  the  system’s  functional  relationships] 

System  Assessment  Plan  Background:  [Background  should  begin  with  a 
definition  of  the  problem.  A  system  being  assessed  is  defined  in  terms  of  the  users 
and  materiel.  The  system  could  also  be  described  as  a  concept,  set  of  tactics,  or  other 
abstract  system.  Regardless  of  the  system  type,  the  author  should  address  four  main 
ideas:  the  system  and  its  purpose,  the  systems  position  in  the  hierarchy  of  systems, 
the  systems  boundaries,  and  the  systems  functional  relationships.] 
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II.  Evaluation  Framework.  This  section  contains  the  evaluation 
questions  and  their  corresponding  standards  and  Measures.  The 
Evaluation  Framework  Hierarchy  shows  the  relationship  between  the 
Critical  Operational  Issues  and  Measures  of  Effectivenss  (MOE), 
Performance  (MOP),  Suitability  (MOS),  and  Survivability  (MOSur). 

•  OE:  Is  the  Operational  Effectiveness  of  the  [system]  adequate  to 
achieve  a  score  of  at  least  80  out  of  100?1 

-  OS:  Is  the  Operational  Suitability  of  the  [system]  adequate  to 
achieve  a  score  of  at  least  80  out  of  100  when  Performance  and 
Survivability  are  held  constant  at  threshold  levels?2 

-  OSur:  Is  the  Operational  Survivability  of  the  [system]  adequate 
to  achieve  a  score  of  at  least  80  out  of  100  when  Performance  and 
Suitability  are  held  constant  at  threshold  levels?  Does  [the  system] 
have  the  appropriate  Information  Assurance  (IA)  controls  in  place  to 
ensure  its  Operational  Survivability?3 

COI-X: 

MOE-X:  ◄ - 

MOP-X: 

MOS-X: 


Table  1,  Issues  and  Screening  Criteria,  completes  the  Evaluation 
Framework.  The  Issues  (Evaluation  Questions)  cover  areas  that  may 
not  be  directly  measurable  in  a  mission  profile  and  might  otherwise  go 
unexamined  in  the  course  of  the  evaluation  if  not  considered  before 

IOT. 

MCOTEA  uses  screening  criteria  to  simplify  the  evaluation  process. 
Screening  criteria  reduce  the  number  of  Issues  that  must  be  evaluated 
to  a  more  manageable  level  and  serve  as  binding  constraints  in  system 
evaluation.  A  system  must  meet  its  screening  criteria  to  be  OE,  OS,  and 
OSur. 


13The  conclusions  for  OE/OS/OSur  are  a  direct  result  of  normalizing  mission  results 
from  the  COI  to  a  common  scale,  the  Mission  Capability  Level.  MCL  is  not  a 
determination  required  by  law  or  directive,  but  is  a  systematic  means  MCOTEA  uses 
to  arrive  at  the  required  conclusions  for  OE/OS/OSur.  MCL  is  used  to  assess  how  well 
Marine  operators  using  a  system  can  be  expected  to  fulfill  their  intended  mission  in  a 
realistic  environment.  See  the  Decision  Model  section  of  this  plan  for  further  details. 


Section  II,  Evaluation 
Framework,  contains  the 
evaluation  questions  and  their 
corresponding  standards  and 
Measures.  The  text  in  the 
template  is  boilerplate. 


Important  to  understand  is 
that  the  indention  of  Measures 
under  the  COI  does  not  imply  a 
roll-up  of  Measure  results.  The 
indention  signifies  that  an  MOE 
decomposes  into  other  types  of 
Measures. 


When  writing  a  SAP:  because 
a  SAP  assesses  a  program  at  less 
than  mission  level,  the  template 
for  a  SAP  does  not  include 
COIs,  Screening  Criteria,  or  the 
language  for  OE/OS/OSur. 

When  fewer  than  10  Issues  are 
used  in  a  SAP,  a  simple  list  of 
Issues  and  Measures  suffices. 
Include  References  and  Test 
Event  information  in  paragraph 
format.  If  the  SAP  requires  more 
than  10  Issues,  a  modified  Issues 
and  Screening  Criteria  table 
is  used  (no  columns  for  Issue 
Category,  COIs  Affected,  or 
Screening  Criteria). 
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The  Evaluation  Framework 
continues  with  a  brief  explanation 
of  the  columns  in  table  l.This 
text  is  boilerplate.  Following  this 
standard  language  is  table  1  itself, 
depicted  on  the  next  page. 

Notes  on  column  1:  When  listing 
Issues,  allow  them  to  flow  in 
sequential  order.  If  an  Issue  is 
subsequently  dropped,  do  not 
renumber  the  other  Issues. 

Notes  on  column  4,  Measures: 

The  method  of  measurement 
should  correspond  with  the 
standard  identified  within  the 
Issue.  For  example,  if  the  Issue 
states,  “Can  the  system  be 
emplaced  in  less  than  5  minutes?” 
then  the  corresponding  Measure 
should  be  a  time-based  metric, 
such  as  “Emplacement  Time 
(minutes).”  However,  the  Measure 
should  not  contain  extraneous 
information  that  detracts  from 
the  Measure  itself  and  should  not 
identify  the  conditions  for  the 
test.  If  no  Measure  is  applicable 
(e.g.,  certification  or  verification 
are  being  called  for,  not  a 
standard  Measure),  use  “N/A.” 

Notes  on  column  6,  COIs/ 
Measures  Affected: 

When  a  Measure  is  first  listed 
under  a  COI  in  the  Evaluation 
Framework  and  will  also  be 
used  to  satisfy  an  Issue  in  table 
1,  cite  the  Measure  in  column  6. 
Doing  so  supports  traceability 
and  assures  the  reader  that  a 
Measure  supporting  both  a 
COI  and  an  Issue  is  not  being 
double-counted. 


This  section  provides  a  brief  explanation  of  the  purpose  of  each  column 
in  the  Issues  and  Screening  Criteria  table. 

Column  1.  Issue  Number.  The  test  team  derives  Issue  numbers  from 
the  Operational  Task  Analysis.  The  numbering  system  is  meant  to  be 
simple  and  logical  for  the  design  of  each  test. 

Column  2.  Issue  Category.  Each  Issue  is  categorized  as  OE,  OS,  or 
OSur  as  the  first  step  in  detemining  what  effect,  if  any,  the  Issue  will 
have  on  the  OE,  OS,  and  OSur  conclusions. 

Column  3.  Issue  Description.  States  the  evaluation  question.  Issues 
deriving  from  Attributes  with  thresholds  are  marked  with  an  asterisk. 
Issues  without  standards,  but  of  interest  to  the  evaluation,  are  stated 
as  open-ended  questions;  in  other  words,  the  Issue  identifies  the 
dimensions  of  measure  but  not  the  level  of  satisfaction.  Issues  annotated 
with  an  asterisk  denote  Attributes  with  thresholds  that  must  be 
examined  by  MCOTEA.  (*  =  Threshold) 

Column  4.  Measures.  Documents  the  measurements  that  must  be  taken 
to  answer  the  individual  Issues.  Measures  that  support  COIs  can  also 
support  Issues. 

Column  5.  Reference  for  Standard.  Traces  the  standard  to  its  source. 
Also  notes  KPPs  when  applicable. 

Column  6.  COIs  and  Measures  Affected.  Identifies  the  applicable  COIs 
for  an  Issue.  An  Issue  can  apply  to  one,  more  than  one,  or  all  COIs. 

Column  7.  Screening  Criteria.  Indicates  that  an  Issue  will  in  some 
way  constrain  an  answer  to  OE,  OS,  or  OSur.  “Yes”  in  this  column 
means  that  an  Issue  is  a  screening  criterion  and  will  therefore  become  a 
binding  constraint  on  the  evaluation  answer,  depending  on  its  outcome. 
“No”  in  this  column  means  that  the  Issue  is  not  a  screening  criterion. 
Refer  to  chapter  3-1  for  a  detailed  discussion  of  this  topic. 

Column  8.  Test  Events.  Indicates  the  test  events  that  will  yield  data  on 
the  issue  for  MCOTEA’s  evaluation  of  the  system. 


Further  essential  information  about 
Screening  Criteria  (column  7): 

Issues  that  require  investigation 
over  the  course  of  the  system’s 
development  but  are  also  considered 
in  the  analytic  model  are  typically 
answered  “No.”  An  example  is  a 
Reliability  Issue,  which  usually  has  a 
threshold  value  that  must  be  reported 
out.  However,  due  to  the  Reliability 
parameter’s  nature  and  its  obvious 
effect  on  mission  outcomes,  the 
threshold  value  will  most  likely  be 
incorporated  in  the  analytic  model. 


An  Issue  identified  as  “No”  in  the 
Screening  Criteria  column  might 
not  affect  the  final  evaluation  via 
the  screening  criteria  or  the  analytic 
model.  These  would  typically  be 
lower-level  Issues  that  address 
component  specification,  early 
maturity  parameters,  etc.,  which  have 
little  relevance  when  determining  OE, 
OS,  and/or  OSur,  but  may  be  relevant 
to  earlier  stages  in  the  evaluation. 
These  issues  are  used  by  the  Program 
Office  to  ensure  the  system  is  ready 
for  IOT. 
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Table  1.  Issues  and  Screening  Criteria 


Issue 

No. 

Issue 

Category 

OE,  OS,  OSur) 

Issue  Description 
(*=threshold) 

Measures 

Reference 

for  Standard 

COIs/ 

Measures 

Affected 

Screening 
Criteria  (Y/N) 

Test  Events 

I-x.x 

OE 

Can  the  system  be  emplaced 
in  less  than  5  minutes*? 

Emplacement 
Time  (minutes) 

CDD  Para. 

6.2.3 

coil 

Y 

MCSC 

LUE 

I-x.x 

OS 

Is  the  MTBF  of  the  system 
greater  than  or  equal  to  250 
hours*? 

MTBF  (hours) 

CDD  Para. 

12.3.1 

(KPP) 

All 

N 

DT-2, 

MCSC 

LUE,  and 
OA 

I-x.x 

OS 

Has  the  J-6  interoperability 
certification  been  obtained? 

CDD  Para. 

14.2 

All 

Y 

N/A 

etc. 

OSur 

Has  the  system  achieved  an 
evaluation  score  of  at  least  80 
on  protect,  detect,  respond, 
and  restore  IA  controls 
implementation? 

I A  score  (0-100) 

CDD  Para. 
13.5.3 

All 

Y 

DT  IV&V 

COIs  Affected  and  the  Screening  Criteria 
columns  become  clear  when  determining 
the  extent  to  which  Issues  will  constrain 
evaluation  answers.  (This  process  considers 
only  “Yes”  Issues.)  For  Issues  that  apply 
to  one  or  more  COIs,  but  not  all  COIs, 
the  evaluator  uses  a  flow  chart  process 
that  constrains  the  answer  to  the  COIs  in 
column  6.  The  constraint  of  these  Issues 
still  allows  the  Mission  Capability  Level 
to  drive  the  answer  to  the  extent  that  the 
unaffected  COIs  are  satisfied  using  the 
analytic  model  process. 

Essentially,  this  constraint  implies  that  the 
Issue  affects  some  but  not  all  mission  areas 
of  the  system  under  evaluation.  Therefore, 
the  degree  of  threshold  satisfaction  can 
influence  only  certain  aspects  of  the 
evaluation.  For  example,  certain  missions 
may  require  systems  to  satisfy  an  interface 
and  subsequent  information  exchange 
requirement  to  successfully  complete 
a  certain  mission  type.  However,  this 
interface  may  not  be  required  for  all 
mission  types;  therefore,  the  evaluation 
should  only  penalize  the  discrete  mission 


types  and  not  globally  penalize  the  system 
when  evaluating  mission  areas  not  needing 
that  requirement. 

If  Issues  with  “Yes”  in  the  Screening 
Criteria  column  and  “All”  in  the  COIs 
Affected  column  fail  to  satisfy  the  criteria, 
then  they  will  globally  affect  the  evaluation, 
using  a  second  flow  chart  process.  This 
second  process  directly  affects  OE,  OS, 
and  OSur  determinations  regardless  of 
the  MCL  outcomes.  Essentially,  these 
global  screening  criteria  circumvent  the 
MCL  process  entirely  and  constrain  the 
evaluation. 

Notes  on  column  8,  Test  Events: 

The  Test  Events  column  does  not  need 
to  be  completed  before  the  TEMP 
submissions.  However,  after  the  TEMP 
is  complete,  this  table  must  be  updated 
with  test  event  information  to  properly 
map  evaluation  questions  to  data  sources. 
Once  this  table  is  complete,  the  test  team 
uses  the  SEP  as  a  roadmap  to  guide  the 
evaluation  process  as  the  system  matures. 
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Following  table  1  is  section  III, 
Evaluation  Methods. 

The  Evaluation  Method  is  a 
scientifically  based,  transparent, 
and  repeatable  process  that  allows 
evaluation  results  to  withstand 
scrutiny.  This  section  of  the  SEP 
contains  two  parts: 

Introductory  material:  a  statement 
about  the  overall  qualities  of 
the  chosen  evaluation  method 
that  support  transparency  and 
reproducibility. 

Model  building:  the  particular 
type  of  model  to  be  used  in  this 
evaluation  method. 

♦  Screening  Criteria  (binding 
constraints) 

♦  Aggregation  Method  (when  an 
evaluation  contains  more  than  one 
COI) 

♦  Analytic  Model  (focused  on  COIs, 
sourced  from  Issues,  fed  by  data 
obtained  in  testing,  and  incorporates 
performance,  suitability,  and 
survivability) 

♦  Decision  Model  (determines  how 
MCL  will  be  decided  when  linked 
to  the  analytic  model  for  COIs) 

The  sample  depicted  here  illustrates 
the  first  page  of  section  III.  Note: 
a  “benchmark”  equation  as  seen 
in  this  sample  is  not  a  standard 
requirement. 


III.  Evaluation  Methods 


a.  Analytic  Model.  The  analytic  model  describes  the  system  in  terms  of  parameters 
linked  to  determine  the  level  of  effect.  The  parameters  for  the  model  are  derived  from  the  MOPs 
and  MOSs  [and  MOSurs]  defined  in  section  1 .  [Each]  COI  has  a  unique  Analytic  Model. 

COM .  Can  Marine  Corps  users  access  and  store  information  via  MCEITS  applications,  services, 
and  data? 


The  core  function  of  MCEITS  is  to  host  Marine  Corps  applications  and  data  following  a  net- 
centric  model.  Individual  applications,  both  current  and  legacy,  have  various  storage  and 
operating  environment  requirements,  creating  a  need  for  a  large  infrastructure  to  support  that 
capability  as  well  as  the  capability  to  expand  for  future  applications  and  services  essential  to  the 
Marine  Corps.  Fundamentally,  a  net-centric  system  such  as  MCEITS  is  required  to  be  folly 
functional  and  online  at  all  times.  Also  integral  to  net-centric  hosting  of  Marine  Corps 
applications,  services,  and  data  is  the  integrated  test  and  system  readiness  certification  of 
applications,  continuous  self-monitoring,  and  a  provision  for  a  “help  desk”  used  in 
troubleshooting  problems  end-users  may  encounter. 

The  obvious  pitfall  of  a  net-centric  system  such  as  MCEITS  is  the  inherent  dependency  upon  the 
connectivity  hardware  that  exists  between  a  user  and  the  system  boundary.  MCEITS  has  no 
control  over  the  performance  of  the  network  that  allows  users  to  access  MCEITS,  but  all  users 
are  forced  to  use  that  network  in  order  to  touch  MCEITS.  To  mitigate  the  impact  of  non- 
MCEITS  hardware  on  the  scoring  of  MCEITS  in  this  SEP,  a  benchmark  scoring  system  will  be 
used.  The  benchmark  score  will  eliminate  the  negative  effects  of  slow  computing  environments 
as  well  as  slow  network  connections  by  providing  a  “gold  standard”  against  which  a  typical 
user’s  communication  with  MCEITS  can  be  compared  (ref.  a). 


Each  benchmark  test  will  follow  precise  test  scripts  and  will  be  robust,  repeatable,  and 
representative  of  a  typical  organic  or  application  thread. 


S 


CT, 

CTa 


(Benchmark) 


Where 

S  =  Benchmark  score 

CT  =  Communication  time,  subscripted  for  Actual  and  Ideal  times 


Communication  Time  is  defined  as  the  time  spent  by  the  user’s  computing  environment 
communicating  over  the  network  cloud  with  MCEITS.  This  captures  the  latency  of  the  network 
between  the  user  and  MCEITS.  Each  benchmark  will  be  run  on  a  computer  that  meets  exact 
specifications  and  has  a  known  background  task  workload,  keeping  the  user’s  computing 
environment  as  a  constant.  This  benchmark  score  will  be  used  in  the  calculation  of  this  Analytic 
Model. 
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MOE-l:  Probability  of  Functional  Thread  Completion 

The  Probability  of  Functional  Thread  Completion  is  defined  as  the  probability  that  any  user  of 
MCEITS  will  be  able  to  complete,  from  start  to  finish,  their  functional  thread  using  MCEITS. 
This  can  be  any  mission  for  which  MCEITS  is  capable  of  providing.  The  Probability  of 
Functional  Thread  Completion  can  be  found  by  multiplying  together  the  probabilities  of 
completing  each  sub-step  within  a  functional  thread: 

P[FTC]  =  P[Access]  *  P[Task  Completion]  (1) 

Each  probability  is  derived  below  using  MOPs. 


MOP-1:  Probability  of  Access 

The  probability  that  any  user  will  be  able  to  access  a  given  application  on  MCEITS  takes  into 
account  the  probabilities  that  MCEITS  and  the  application  will  be  available,  that  MCEITS  has 
not  reached  maximum  capacity,  and  that  both  MCEITS  and  the  hosted  application  can  properly 
provide  access  to  the  application  itself.  This  is  described  by  the  relationship  in  equation  2: 

P{Access }  =  P{MCEITS  Connection]  *  P{Application  Access]  (2) 

Each  component  of  Equation  2  is  derived  and  described  in  further  detail  below. 

MOP-1.1:  Probability  of  MCEITS  Connection 

The  probability  that  a  user  will  be  able  to  connect  to  MCEITS  at  any  given  point  in  time  is 
governed  by  the  relationship: 

P{MCEITS  Connection]  =  Aomcelts  *  AoappUcation  *  P [Connectivity]  (3) 

Each  subcomponent  of  P  {MCEITS  Connection}  is  derived  below. 


MOS-1.1.1:  Availability 

Availability  (A)  is  the  probability  that  the  system  will  be  able  to  perform  its  mission  when  the 
mission  is  called  for  at  a  random  point  in  time  (ref.  b).  Availability  is  determined  by  the 
proportion  of  time  MCEITS  is  in  an  operable  state: 


A 


mceits 


MTBF 

MTBF  +  MTTR 


(4) 


Availability  for  hosted  applications  (Aappijcation )  will  be  defined  by  SLAs.  MTBF  and  MTTR  are 
described  below. 


MOS-1. 1.1.1:  Mean  Time  Between  Failures  (MTBF) 

This  Measure  applies  only  to  the  Up  Time  between  failures  at  the  EITC  that  deny  outside  access 
to  a  user.  Maintenance  Actions  may  be  performed  on  any  part  of  MCEITS  and  still  be  considered 
Up  Time  if  and  only  if  an  outside  user’s  thread  is  not  denied  completion  by  the  Action,  e.g., 
redundant  server  replacement. 


Section  III  continues  with 
formulas  for  the  Measures 
as  depicted  in  this  sample. 
Formulas  are  generated  with 
Microsoft®Equation  Maker  and 
placed  in  a  one-line  table  for 
ease  of  numbering.  The  table  is 
included  in  the  SEP  template. 


Referencing  Equations: 

MCOTEA  requires  equations  to 
be  referenced  when  they  are  not 
the  original  work  of  MCOTEA 
analysts.  The  sample  pages  shown 
here  do  not  reflect  referenced 
equations,  so  the  following 
examples  are  provided: 

R=e(ni‘i/mtbomd  woui(j  need  a 
reference  from  an  applicable 
RAM  source  because  it  is  a 
standard  equation  the  analyst  is 
using  in  a  MCOTEA  document. 

However,  P  *P.*A  =P  does  not 
need  a  reference  because  it  is  an 
original  derivation  of  mission 
accomplishment  specifically 
developed  for  a  system  by 
MCOTEA  analysts. 

The  SEP  template  on  the 
shared  drive  contains  a  sentence 
in  section  III  that  accounts 
for  unreferenced  equations: 
“Equations  without  references 
have  been  developed  by 
MCOTEA  to  support  system 
analysis.” 
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The  Decision  Model  section  uses 
narrative,  tables,  and  graphs  to 
convey  model  information. 

Guidance  for  creating  tables, 
graphs,  and  charts  can  be  found 
at  the  end  of  chapter  3-6. 


c.  Decision  Model.  Mission  Capability  Level  (MCL)  is  the  output  of  the  decision  model 
when  linked  to  the  analytic  model  for  a  COI.  Because  the  analytic  model  includes  system 
effectiveness  and  suitability  in  the  mission  context,  the  evaluator  is  able  to  draw  the  necessary 
conclusions  regarding  OE,  OS,  and  OSur.  For  standardization  purposes,  MCL  is  further  defined 
as  a  score  on  a  continuous  scale  from  0  to  100,  with  0  being  the  lowest  possible  score  and  1 00 
the  highest.  Table  7  breaks  out  the  MCL  ranges  (ref.  j). 


Table  7.  Mission  Capability  Level 


MCL 

Range 

Fully  Mission  Capable 

The  system,  in  the  mission  context,  has  achieved  at  least  the  equivalent  of  threshold 
performance  or  better. 

80 

100 

Partially  Mission  Capable 

The  system  is  at  least  as  good  as  the  current  capability  but  falls  short  of  the  threshold. 

50 

<80 

Not  Mission  Capable 

The  system  does  not  improve  on  current  mission  capabilities.  The  range  for  Not  Mission 
Capable  is  expanded  from  0<50  to  0<80  when  no  current  mission  capability  exists  for  the 
missions  the  system  is  designed  to  address. 

0 

<50 

[MCOTEA  will  determine  MCL  by  using  a  piecewise  linear  function  for  each  COI  that  equates 
MOE  and  MOS  results  from  the  COI  to  MCLs.] 

COI  Value  Function.  MCOTEA  will  determine  MCL  by  using  a  piecewise  linear  function  for 
COI-1  that  equates  MOE  results  for  the  COI  to  MCLs.  The  data  points  used  to  construct  the 
functions  for  the  COI  appear  in  table  5.  MCOTEA  will  update  table  5  once  the  current  capability 
levels  have  been  assessed.  This  SEP  will  be  updated  as  new  information  becomes  available  to 
reflect  the  thresholds  for  MCEITS  MCL  more  accurately. 


Figure  3.  OE  Mission  Capability  Level  Piecewise  Function 
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IV.  Assumptions 

a.  The  SEP  is  a  living  document  that  identifies  gaps  in  understanding  where  requirements  are  not  fully 
delineated.  As  letters  of  clarification  are  resolved,  MCOTEA  will  update  the  SEP  to  reflect  the  requirement.  For 
example,  based  on  the  CPD  requirement  for  IOC,  threshold  values  are  set  to  10  percent  of  objective  capability 
unless  otherwise  stated  within  the  CPD.  Threshold  values  are  to  be  determined  by  Deputy  Commandant, 
Combat  Development  and  Integration. 

b.  Much  of  the  required  information  for  this  evaluation  is  as  yet  unknown.  PM  MCEITS  and  HQMC 
have  not  yet  decided  which  application  will  or  will  not  be  hosted  on  MCEITS,  so  information  about  them  and 
their  Service-Level  Agreements  is  unavailable.  The  list  provided  above  is  notional  and  cannot  be  used  officially 
at  the  time  of  writing  this  SEP.  MCOTEA  will  update  this  SEP  and  any  subsequent  Test  Plan  to  reflect  all  new 
data  as  it  arrives. 

V.  Limitations 

c.  The  evaluation  of  Operational  Survivability  is  not  wholly  based  on  operational  employment  in  a 
representative  threat  environment.  The  evaluation  of  OSur  is  based  on  the  implementation  of  and  compliance 
with  Information  Assurance  controls. 

3.  Conclusion.  MCOTEA  will  use  this  SEP  as  a  basis  for  observing  DT  events,  contributing  to  further  test 
documentation,  and  reaching  a  final  conclusion  about  OE/OS/OSur. 

4.  References 

a.  Gunther,  Neil  J.  Benchmarking  Blunders  and  Things  That  Go  Bump  in  the  Night.  Paper  presented  at  the 
Workshop  on  Software  Performance  and  Reliability,  Menlo  Park,  CA,  Apr  2004. 

http :  //  www  .cmg.org/  measure  it/issues/ mit3  2/m_3  2_2 .  html 

b.  Department  of  Defense.  Guide  for  Achieving  Reliability,  Availability,  and  Maintainability.  Aug  2007. 

Annex  A.  FD/SC  Charter  (when  it  becomes  available) 


The  SEP  template  finishes  with  a 
diagram  of  the  complete  evaluation 
process  as  all  elements  of  the  model 
come  together. 


Assumptions  and  Limitations  are 
written  for  the  particular  system 
being  evaluated.  The  Conclusion 
paragraph,  however,  is  boilerplate. 
Note  that  the  numbering  of 
the  Conclusion  and  References 
paragraphs  returns  to  the  base 
template. 


References  should  be  used 
throughout  the  text  and  are  listed 
in  section  4  in  order  of  appearance. 
See  the  end  of  this  chapter  for 
guidance  on  creating  and  citing 
references. 
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Developmental  Test 
Observation  Plan 

Author:  OTPO/Test  Manager,  with 
possible  OA  assistance 

The  DT  Observation  Plan  is 
generally  short  (2  pages  or  so)  and 
focuses  on  a  particular  DT  event 
listed  in  the  SEP.  (See  chapter  3-3 
for  a  full  discussion.) 

Paragraph  5,  Evaluation 
Questions,  is  the  heart  of  the  DT 
Observation  Plan.  The  evaluation 
questions  are  taken  directly  from 
the  SEP  or  SAP.  The  exception 
to  this  is  observation  of  an  early 
technology  demonstration,  before 
the  SEP  or  SAP  is  written.  In 
that  case  MCOTEA  observes  the 
event  for  system  familiarization 
and  will  not  analyze  any  data  from 
the  event. 

If  no  SEP  or  SAP  is  available 
yet,  restate  test  objectives  and/or 
threshold  requirements  from  the 
DT  Plan. 

The  DT  Observation  Plan  is 
approved  and  signed  at  the 
Division  level. 


Developmental  Test  Observation  Plan 
[System  Name] 


1.  Purpose.  [State  purpose  of  document,  name  of  event,  date  and  location  of 
event.  Follow  with  purpose  of  event  itself  and  MCOTEA’s  precise  purpose  for 
being  there.  For  early  events  such  as  Technology  Demonstrations,  MCOTEA’s 
purpose  is  to  gather  information  that  will  aid  in  planning  future  integrated 
testing.] 

Sample: 

This  document  describes  MCOTEA’s  plan  for  observing  the  Theater  Battle 
Management  Core  System  (TBMCS)  Maintenance  Release  2  (MR2) 
Developmental  Test  (DT)  scheduled  for  14  February-11  March  2011  at 
the  Idaho  National  Laboratory  in  Idaho  Falls,  ID.  This  multi-Service  DT 
event,  led  by  the  46th  Test  Squadron  of  the  U.S.  Air  Force,  will  test  the 
interoperability  and  functionality  ofTBMCS  spiral  1.1.3  MR2  and  evaluate 
its  ability  to  meet  government  requirements  in  preparation  for  Operational 
Test  (OT)  in  August  2011.  MCOTEA  will  observe  test  events  from  14-26 
February  to  determine  the  extent  to  which  the  Test  Plan  is  followed  and  that 
data  collection  is  comprehensive  and  complete. 

2.  Background.  [Provide  the  problem  definition  (capability  gap)  and  a 
brief  (one  paragraph)  system  description.] 

3.  Schedule.  [State  the  test  event  schedule  from  the  DT  Plan,  if 
available.] 

4.  Organization.  [State  the  billets  of  the  members  of  the  observation  team 
(no  names);  who  is  conducting  the  DT  event  (contractor,  government, 
etc.);  who  else  from  the  Program  Office  may  be  attending  the  event,  etc. 

5.  Evaluation  Questions.  [Connect  the  DT  event  with  Issues  from 
the  SEP;  e.g.,  a  Logistics  Demonstration  event  could  be  used  for  a 
Supportability  Issue.  Identify  the  Attribute  thresholds  that  will  be 
examined  by  the  test,  if  any.  Cite  the  section  of  the  DT  Plan  being 
referenced.  Finish  with  statement  about  the  date  MCOTEA  expects  to 
receive  the  post-event  DT  Report.] 

6.  References.  [DT  Plan,  MCOTEA’s  SEP/SAP  (do  not  reprint)  plus 
references  used  in  the  text.  Do  not  cite  general  (background)  references.] 

Annex  A.  Data  Collection  Forms  (can  be  simple  tables) 

Annex  B.  Incident  Response  Plan  (use  template  in  DT  Observation  Plan 
folder) 
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Developmental  Test  Observation  Report 
[System  Name] 

1.  Purpose.  [State  the  purpose  of  this  document  (to  provide  MCOTEA’s 
observations  of  DT  event  execution)  and  the  purpose  of  the  event  itself.] 

2.  Background.  [Restate  the  system  description  from  the  Observation 
Plan.] 

3.  Scope.  [Scope  of  the  report  is  what  the  observer  saw  of  test  conduct, 
without  analysis  or  conclusion.] 

4.  Objective.  [“The  objective  of  this  report  is  to  formally  record 
MCOTEA’s  observations  of  test  execution  from  the  event  before 
receiving  the  DT  Report.”] 

5.  Assumptions,  [if  any;  brought  forward  from  the  SEP)  Example: 
MCOTEA  assumed  that  the  system  under  test  had  reached  a  certain  level 
of  maturation  by  the  time  of  the  event.  State  any  issues  that  may  have 
been  identified  in  previous  testing  that  have  not  been  resolved.] 

6.  Limitations,  [of  this  report.  State  that  this  report  cannot  evaluate  test 
results  without  the  Test  Report  itself.] 

7.  Methods.  [Method  of  observation,  such  as  tracking  DT  Plan  test 
threads,  operator  surveys,  etc.,  or  analytical  method  of  evaluation.  Include 
the  qualitative  characteristics  of  test  conduct.] 

8.  Results,  [of  observing  test  execution.  Discuss  by  Evaluation  Question 
or  Test  Objective  if  Evaluation  Questions  were  not  used.  If  deviations 
from  the  DT  Plan  occurred,  discuss  them  in  detail.] 

9.  Insights.  [Preface  any  statements  here  with  “It  appears  that”  something 
about  system  performance  may  bear  further  watching;  statement  must  be 
nonjudgmental.  Purpose  is  to  make  the  PM  aware  of  potential  risk  areas. 
Also  highlight  positive  areas  when  notable.] 

10.  Recommendations.  [State  only  recommendations  for  further  or 
repeat  testing  based  on  insufficiency  of  test  planning  or  execution;  for 
example,  an  Issue  not  addressed  or  a  threshold  not  examined.] 

11.  References.  [Cite  DT  Plan  and  Observation  Plan;  do  not  append  the 
references  to  this  report.] 

Annex  A.  Observation  Notes  for  the  Record  (supporting  observation 
data) 


DT  Observation  Report 

Author:  OTPO/TM 

The  OTPO  or  Test  Manager 
writes  the  DT  Observation  Report 
immediately  after  returning  from 
the  DT  event.  The  process  assumes 
that  the  OTPO/TM  have  not  yet 
received  the  expected  DT  Report, 
meaning  that  only  test  execution, 
not  system  performance,  can 
be  discussed  at  this  point.  The 
Division  Head  sends  a  copy  of  this 
report  within  10  working  days  of 
event  completion  to  the  PM. 

Note:  When  MCOTEA  is  not 
evaluating  results  from  an  event, 
the  DT  Observation  Report  should 
state  this.  The  OTPO/TM  should 
copy  the  OA  and  keep  the  report 
on  file  for  reference. 

The  DT  Observation  Report 
is  approved  and  signed  at  the 
Division  level. 
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Intermediate 
Assessment/System 
Assessment  Report 

Author:  OA 

IARs  are  produced  throughout 
the  span  of  MCOTEA’s 
involvement  with  Integrated 
testing  and  are  usually  based 
on  a  SEP.  They  present 
MCOTEA’s  evaluation  of  test 
results  for  use  at  Gate  Reviews 
and  ultimately  contribute  to 
final  system  evaluation.  The 
OA  prepares  the  report  after 
receiving  DT  concurrence 
from  the  OTPO/TM  or  after 
analysis  of  a  MCOTEA-led 
event.  Reports  may  be  prepared 
individually  or  in  aggregate.  The 
audience  for  the  report  is  the 
PM  and  the  MDA. 

A  SAR  is  based  on  a  SAP  and 
is  a  final  document,  prepared 
by  the  OA  after  all  information 
is  obtained  and  analyzed  in 
support  of  an  AAP,  ACAT IV 
(M),  or  other  non-program  of 
record.  The  audience  for  the 
report  is  the  PM  and  MDA. 


Intermediate  Assessment/System  Assessment  Report 
[System  Name] 

1.  Purpose.  This  Intermediate  Assessment  Report  presents  MCOTEA’s 
evaluation  of  test  results  from  the  [event,  date,  location] .  This  report  is 
intended  for  the  PM  and  MDA’s  use  [at  a  Gate  Review  or  other  purpose].  At 
the  conclusion  of  planned  system  testing,  MCOTEA  will  aggregate  the  results 
presented  here  with  those  of  other  developmental  and  operational  tests  to 
determine  final  system  evaluation. 

For  a  SAR:  This  System  Assessment  Report  presents  MCOTEA’s  evaluation  of  test 
results  from  the  [event,  date,  location].  This  report  addresses  evaluation  questions 
from  the  System  Assessment  Plan  and  is  a final  document  intended  for  the  PM  and 
MDA’s  use. 

2.  Background.  [State  problem  definition  and  system  description.] 

3.  Scope.  This  report  evaluates  test  results  from  [test  event]  only  and  is 
not  intended  to  determine  OE/OS/Sur  or  to  be  a  comprehensive  system 
evaluation. 

4.  Objective.  This  report’s  objective  is  to  present  unbiased  evaluation  of  test 
results. 

5.  Assumptions.  [Bring  forward  from  SAP/SEP  and  other  individual  tests  as 
applicable.] 

6.  Limitations,  [of  this  evaluation,  based  on  test  deviations  or  inherent  limits 
from  the  SAP/SEP] 

7.  Methods.  [State  the  analytical  method  of  the  evaluation.] 

8.  Results.  [For  an  IAR:  Summarize  data  results  that  highlight  risk  areas 
based  on  evaluation  questions  examined  or  how  the  system  is  maturing  based 
on  satisfying  the  evaluation  questions.  For  SAR:  Summarize  data  results  from 
evaluation  questions  of  the  system.] 

9.  Insights.  [State  any  verifiable  trends  supported  by  test  results,  positive  or 
negative,  that  the  assessment  reveals.] 

10.  Conclusions.  [State  the  overall  summary  of  evaluation  questions  without 
repeating  data.  Do  not  introduce  any  new  ideas  or  say  anything  not  already 
discussed  in  the  text.] 

11.  Recommendations.  [State  any  improvements,  mitigation,  or  follow-on 
testing  needed  for  the  system.  Recommendations  flow  from  ideas  in  Results, 
Insights,  and  Limitations.] 

12.  References,  [as  appropriate:  SAP/SEP;  DT  Plan;  MCOTEA  DT 
Observation  Plan;  MCOTEA  Test  Report;  DT  Report.  Do  not  reprint  or 
append  any  references.] 


Annex  A.  Analytic  Results 
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[Type  of  ]  Test  Plan 
[System  Name] 

1.  Purpose.  [Use  language  such  as  the  following:  This  Initial  Operational 
Test  (IOT)  Plan  provides  test  execution  and  management  guidance  for  the 
[system].  MCOTEA  will  use  data  obtained  from  the  IOT,  along  with  other 
data  collected  during  integrated  testing,  to  prepare  an  Operational  Test 
Agency  (OTA)  Evaluation  Report  (OER),  which  will  provide  conclusions 
concerning  the  OE,  OS,  and  OSur  of  the  [system]  based  on  the  Issues  and 
Measures  contained  in  this  plan.  The  conclusions  will  be  used  to  support  a 
United  States  Marine  Corps  Milestone  C  LRIP  decision  for  the  [system]]. 
[For  EOAs  and  OAs,  state  that  the  data  will  be  used  to  evaluate  system  progress 
and  to  provide  potential  insight  into  system  trends  or  deficiencies.  For  System 
Assessments,  state  that  the  data  will  be  used  to  examine  the  risks  and  benefits  of  the 
system.] 

2.  Background.  [Provide  the  problem  definition  (capability  gap)  and  system 
description.] 

3.  Schedule.  [Insert  table  that  lists  dates,  events,  locations,  and  POCs.  “Test 
phases”  are  no  longer  necessary.  The  OA  must  provide  the  Trial  Sequence 
before  the  schedule  can  be  completed.] 

4.  Organization.  [Insert  chain  of  command  graphic  here  with  narrative  as 
needed,  explaining  test  team,  local  chain  of  command,  and  other  test  support 
staff.  Adjust  graphic  as  needed  for  individual  test  organization.] 

5.  Assumptions,  [if  any,  brought  forward  from  the  SEP] 

6.  Limitations,  [of  the  test.  The  Test  Limitations  described  here  will  become 
Annex  A  of  the  Test  Report.] 

7.  Executable  Test  Plan.  [This  section  of  a  Test  Plan  displays  the 
information  that  the  test  team  needs  for  successful  test  execution.  The  first 
section  presents  a  global  view  of  data  requirements  and  test  structure  in 
table  format.  The  middle  section  contains  the  test  trials  in  narrative  form. 
Following  the  narrative  is  a  more  detailed  event  schedule  for  the  Test 
Manager’s  use.  The  sample  below  illustrates  how  test  details  are  filled  in. 

This  process  repeats  itself  for  each  COI/Issue.  The  Measure  of  Effectiveness 
is  listed  on  the  first  page  with  its  Issue,  while  Measures  of  Suitability  and 
Performance  appear  before  the  Trial  Conduct  section.  Note  for  Pilot  Test: 
begin  Trial  Sequencing  with  “PT  1,”  for  example,  and  begin  Trial  Conduct 
narrative  with  discussion  of  Pilot  Test.] 

COI-1 :  Can  the  XXXX  system  M-1 :  Probability  of  Identification 
identify  hostile  enemy  actions 
with  at  least  a  0.70  probability  of 
success? 


Test  Plan  Outline 
(page  1  of  2) 

Author:  OTPO/TM,  with 
possible  OA  assistance 

Test  Plans  contain  a  number 
of  standard  tables  (as  found  in 
the  template)  but  few  unique 
graphics  apart  from  the  tables. 
Graphics  that  do  appear  will 
generally  be  maps  or  Trial 
Conduct  diagrams  unique  to 
each  program. 

The  schedule,  located  in  Annex 
A,  must  be  highly  detailed, 
with  hour-by-hour  descriptions 
of  the  measures  being  taken. 

It  must  also  identify  the 
Measures  and  Trials. 
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Test  Plan  Outline 
(page  2  of  2) 


The  arrows  indicate  the  flow 
of  information  development  in 
the  Data  Requirements  section 
of  the  plan. 


Sample  Size  and  Test  Design 

Trials  by  Variable 
Combinations 

Full  Sun 

System  1 

Sniper 

IED 

IDF 

None 

1 

1 

2 

2 

Data  Requirements 

Data  Collection  Method 

Enemy  Action  (none,  sniper,  IED  emplacement) 

Form  X:  Enemy  Action,  Time  of  Day,  System  ID,  and 

Operator  ID  will  be  preloaded  for  each  planned  trial 
for  the  Data  Collectors  at  the  start  of  each  trial 

Indirect  Fire  (mortar) 

Time  of  Day  (Full  Sun,  Dusk/Dawn,  Night) 

Data  Reduction 

Data  Analysis  Method 

Filter  the  records  by  system  ID 

Categorical  factors  including  Enemy  Action,  Time 

of  Day,  and  System  ID  will  be  examined  using 

Binary  Logistic  Regression  with  alpha  set  to  0.05  to 

Remove  all  records  from  the  data  that  are  not 

determine  if  any  factor  is  a  significant  predictor  of 

identified  as  OpT 

success 

Resource/  Personnel  Quantity 

COC  (RGS,  Radio)  (Critical) 

2 

0PF0R 

25 

Trial  Sequence- System  1 

T est  Day 

Trial  # 

Illumination 

RGS  Status 

Enemy 

Action 

1 

1 

Full  Sun 

On 

IED 

1 

2 

Dark 

On 

None 

Trial  Conduct.  [SAMPLEJAt  the  beginning  of  the  trials  the  COCs  will  have 
their  RGS  monitors  turned  to  the  designated  position  in  accordance  with  the  trial 
sequence.  Just  prior  to  beginning  the  event,  the  Hostile-Rifle/Scope,  Hostile- 
Mortar,  Neutral,  Neutral-Rifle,  Friendly- Rifle/Scope,  and  Friendly-Mortar  teams 
will  be  distributed  to  their  respective  positions.  Only  the  Hostile-Mortar,  Neutral- 
Rifle,  and  Friendly-Mortar  teams  can  be  visible  to  the  towers  at  the  beginning  of 
trial  #1.  [Add  maps,  diagrams,  etc.,  as  required.] 


Annex  A.  Logistics  Summary  [comprehensive  resources  and  highly  detailed  (hour  by  hour) 
daily  master  schedule.  Identify  all  Measures  and  trials.  Use  MS  Word,  not  Excel.] 

Annex  B.  Data  Collection  Forms 

Annex  C.  Safety  Plan  [See  Templates  on  the  shared  drive.] 
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Test  Data  Report 
[System  Name] 

1.  Purpose.  This  Test  Data  Report  provides  raw  and  reduced  test 
results  from  the  [type  of  test]  of  [the  system]  for  an  early,  unanalyzed 
look  at  test  data. 

2.  Background.  MCOTEA  collected  the  data  in  this  report  in 
accordance  the  [type  of]  Test  Plan  (ref.  a). 

3.  Scope.  This  report  is  limited  to  data  from  the  test  MCOTEA 
conducted  on  the  system  in  [location]  from  [dates]  . 

4.  Objective.  The  objective  of  this  report  is  to  make  test  data 
available  for  review  while  MCOTEA  continues  the  evaluative 
process  that  will  lead  to  conclusions  about  Operational  Effectiveness, 
Operational  Suitability,  and  Operational  Survivability. 

5.  Deviations.  [Summarize  deviations  from  the  Test  Plan.  Ensure 
that  any  deviation  that  affects  a  data  element  or  data  set  is  explained 
as  a  caveat  to  the  data.  Explain  deviations  and  caveats  in  detail  in 
Annex  A.] 

6.  Methods.  This  report  presents  test  data  in  electronic  format. 
[Assumes  use  of  CD  for  all  data.  Adjust  if  necessary.] 

7.  Results.  Annex  B  on  the  attached  CD  presents  a  detailed 
breakdown  by  Measure,  in  tabular  format,  of  the  data  obtained  at 
IOT.  An  index  tab  provides  a  link  to  each  labeled  Measure. 

8.  References 

a.  MCOTEA.  \Name  of  Test  Plan]  [Month  Year]. 

Annex  A.  Test  Plan  Deviations 

Annex  B.  Supporting  Data  for  Test  Measures 


Test  Data  Report 
Outline 

Author:  OTPO/Test 
Manager,  with  OA 
assistance 
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Evaluation 
Report  Outline 

Author:  S-2,  with 
OTPO/TM  assistance 


The  ideal  length 

for  an  Evaluation 
Report  is  5-10  pages. 
Evaluation  reports 
require  an  Executive 
Summary,  which  does 
not  exceed  one  page 
and  is  targeted  at  the 
flag/SES  level  (see 
page  4-6  for  more  on 
Executive  Summaries). 
The  main  body  of  the 
report  is  targeted  to  the 
05-06  level  while  the 
annexes  and  appendicies 
are  targeted  toward 
engineers  and  analysts. 


Evaluation  Report 
[System  Name] 

1.  Purpose.  [State  the  type  of  evaluation  report  and  use  language  similar  to 
the  following  example,  adjusting  for  type  of  report.]  “This  OTA  Evaluation 
Report  aggregates  results  from  developmental  test  observation  and  MCOTEA’s 
operational  testing  for  the  [System]. This  report  focuses  on  [acronym  system’s] 
degree  of  mission  accomplishment  and  provides  conclusions  about  Operational 
Effectiveness  (OE),  Operational  Suitability  (OS),  and  Operational  Survivability 
(OSur).  This  document  also  presents  the  results  of  Attributes  with  thresholds  to 
date  from  all  sources  of  testing.” 

2.  Background.  State  problem  definition  (original  capability  gap)  and  system 
description. 

3.  Scope.  This  report  covers  [developmental  and  operational]  testing  results 
accumulated  over  [time  span]. 

4.  Objective.  This  report’s  objective  is  to  present  unbiased  evaluation  of  test  results. 

5.  Assumptions.  [Bring  forward  from  SEP  and  individual  tests.] 

6.  Limitations,  [of  the  evaluation,  based  on  test  deviations  and  inherent  limits 
from  the  SEP] 

7.  Methods.  [State  the  evaluation  method.] 

8.  Results.  [Organize  results  by  COIs  and  how  well  the  mission  is  accomplished 
using  performance/suitability/survivability  characteristics  related  to  COIs.  Place 
detailed  analysis  and  computations  in  Annex  A  (this  includes  IA  or  any  topic 
MCOTEA  analyzes).  State  that  results  of  Attributes  with  thresholds  are  found  in 
Annex  B.  ] 

9.  Insights.  [State  any  unplanned,  verifiable  findings.] 

10.  Conclusions.  [State  the  highest  level  of  conclusion  appropriate  for  the  type 
of  report.  Do  not  introduce  any  new  ideas  and  do  not  include  data.] 

11.  Recommendations.  [State  any  improvements,  mitigation,  or  follow-on 
testing  needed  for  system  based  oft  the  results.] 

12.  References.  [SEP,  Test  Reports,  prior  Evaluation  Reports.  Do  not  annex  or 
print.] 


Annex  A.  Analytic  Results  [append  complete  data  on  CD.  Include  IA  and  other 
topics  for  analysis.] 

Annex  B.  Issues  and  Screening  Criteria  with  Results 
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Request  for 
Clarification  Letter 

Author:  OTPO/TM/  and  OA  as  appropriate 

MCOTEA  participates  in  the  construction 
of  the  capabilities  documentation  through 
the  Capabilities  Documentation  IPT.This 
IPT  presents  an  opportunity  for  MCOTEA 
to  clarify  requirements  in  the  documentation 
early  on.  However,  if  questions  remain  after 
reviewing  a  capabilities  document  and  other 
reference  material  in  detail,  MCOTEA 
writes  a  Request  for  Clarification  letter  to 
DC,  CD&I.  The  purpose  of  the  letter  is  to 
eliminate  ambiguity  and  to  obtain  well- 
defined  requirements.  The  most  productive 
time  to  send  out  a  Request  for  Clarification 
is  anytime  after  receiving  a  draft  or  final 
CDD/CPD.  If  a  newer  edition  of  a  CDD/ 
CPD  is  released,  MCOTEA  may  need  to 
send  out  a  new  letter  (no  limit  exists  on  the 
number  of  letters).  See  sidebar  for  additional 
information. 

The  OTPO/OA  coordinates  with  the 
DC,  CD&I  action  officer  and  the  MCSC 
Program  Manager’s  representative  in 
preparing  this  letter.  In  the  Request  for 
Clarification,  MCOTEA  offers  its  proposed 
interpretation  (or  asks  questions  about 
the  meaning)  of  each  capability  under 
discussion.  In  addition,  MCOTEA  presents 
a  reasonable  interpretation  that  makes 
each  capability  testable  and  resolvable. 

DC,  CD&I,  in  its  response,  concurs  or  not 
with  MCOTEA’s  interpretation.  Where 
it  does  not  concur,  DC,  CD&I  provides 
a  clarified  response  and  other  necessary 
guidance  for  those  items;  DC,  CD&I  is  the 
highest  authority  regarding  the  meaning 
of  capabilities  and  requirements  and  the 
establishment  of  standards. 

The  OTPO/system  evaluator  sends  a 
hard  copy  of  the  Request  for  Clarification 
(standard  naval  letter  format)  to  DC,  CD&I 
as  well  as  through  e-mail,  which  allows 
DC,  CD&I  to  enter  their  responses  directly 
beneath  MCOTEA’s  questions.  The  material 
needing  clarification  is  contained  in  an 
enclosure  to  the  letter. 


MCOTEA  must  watch  for  two  outcomes 
with  a  Request  for  Clarification: 

1.  DC,  CD&I  may  not  concur  with 
MCOTEA’s  interpretation  of  a 
requirement  or  standard 

2.  MCSC  may  send  a  clarification  letter 
that  DC,  CD&I  concurs  with,  which 
may  cause  MCOTEA  to  adjust  its 
interpretation 

In  the  case  of  either  outcome,  the  test 
team  or  OA  may  need  to  adjust  their  plans 
accordingly. 

Using  Citations  in  Text 

When  a  MCOTEA  document  is  being 
reviewed,  it  is  assumed  that  the  contents 
are  the  author’s  original  work  unless  stated 
otherwise.  Borrowing  information  from 
other  sources  to  support  or  supplement 
a  MCOTEA  document  is  perfectly 
acceptable  as  long  as  the  source  is 
referenced.  Words  and  ideas,  also  known 
as  intellectual  property,  are  protected 
by  U.S.  law.  Plagiarism  occurs  when  a 
person  attempts,  intentionally  or  not,  to 
pass  off  another  person  or  organization’s 
intellectual  property  as  his  own.  This  can 
be  avoided  by  properly  citing  sources  in 
the  text  and  in  the  Reference  section  at 
the  end  of  a  document. 

Borrowed  information  can  be 
incorporated  into  a  document  three  ways: 
the  source  can  be  quoted,  paraphrased, 
or  summarized.  When  quoting  a  source, 
the  exact  words  of  the  author  or  speaker 
are  used.  This  includes  information  from 
websites  that  is  copied  and  pasted  into  a 
document.  When  paraphrasing  a  source, 
the  main  idea  is  conveyed  to  the  audience 
while  changing  the  tone,  sentence 
structure,  and  word  choice.  When 
summarizing  a  source,  the  main  idea  is 
conveyed  with  fewer  details  and  the  word 
choice  is  different.  No  matter  how  the 
information  is  incorporated,  the  original 
source  must  be  cited. 

MCOTEA’s  preferred  style  for  citing  a 
source  is  to  place  a  parenthetical  reference 


A  Request  for 
Clarification  Letter 
may  also  be  needed 
when  MCOTEA  must 
derive  standards 
for  evaluation 
questions.  The 
proposed  standards 
are  presented  in 
the  Letter.  MCOTEA 
assumes  concurrence 
for  any  proposed 
standard  for  which 
DC,  CD&I  does  not 
respond. 
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in  the  line  of  text  that  refers  the  reader 
to  the  source  in  the  Reference  list.  A 
document  is  cited  as  follows: 

“The  system  accomplishes  this 
operational  control  through  the 
Regional  Network  Operations 
Security  Centers  by  executing  the  IT 
Governance  and  Information  Assurance 
Frameworks  to  establish  enterprise 
priorities  and  ensure  that  appropriate 
resources  are  allocated  to  resolving 
critical  issues  (ref.  a).” 

The  citation  in  the  Reference  list  is 
styled  as  follows: 

a.  Deputy  Commandant,  Combat 
Development  and  Integration. 

Capability  Production  Document for 
MCEITS  Draft  vl.5.  Mar  2010. 

Websites  are  cited  using  the  date  the 
site  was  last  updated;  however,  if  that 
information  is  not  available,  the  date  the 
information  was  accessed  can  be  used 
instead. 

a.  Parkinson,  Richard.  Traffic 
Engineering  Techniques  in 
Telecommunications.  Infotel 
Systems  Corp:  accessed  April  2011. 
http  ://www.tarrani .  net/  mike/ docs/ 
TrafficEngi  neering.pdf 

If  the  reference  needs  to  be  called  out  in 
the  text,  it  is  written  as  follows: 


“Reference  (a)  indicates  that  this  is 
accomplished  through  the  Regional 
Network  Operations  Security  Center.” 

The  reference  list  appears  at  the  end 
of  a  document  and  provides  detailed 
information  about  the  original  source. 
Each  reference  is  listed  chronologically 
and  labeled  with  a  lower  case  letter  that 
corresponds  to  the  parenthetical  citation 
in  the  text.  The  reference  list  is  based  on 
The  Chicago  Manual  of  Style  format. 

It  is  not  necessary  to  include  sources 
that  pertain  to  the  subject  at  hand  but 
were  not  directly  quoted,  paraphrased,  or 
summarized.  Books  or  articles  that  have 
informed  the  author  but  were  not  used  per 
se  are  not  cited  in  the  References  section. 


References 


Defense  Acquisition  University  (DAU). 

2009.  Defense  Acquisition  Guidebook. 
Virginia:  Defense  Acquisition 
University  Press. 

Department  of  Defense.  1987.  Distribution 
Statements  on  Technical  Documents , 

DODD  5230.24. 

Department  of  Defense.  1995.  Withholding 
of  Unclassified  Technical  Data  from 
Public  Disclosure,  DODD  5230.25. 


SECNAV  (Secretary  of  the  Navy).  2008. 
Implementation  and  Operation  of 
the  Defense  Acquisition  System  and 
the  Joint  Capabilities  Integration 
and  Development  System , 
SECNAVINST  5000.2D. 

United  States  Marine  Corps.  1992.  HQMC 
Supplement  to  the  Department  of 
the  Navy  (DON)  Manual,  MCO 
5216.20. 

University  of  Chicago.  2010.  The  Chicago 
Manual  of  Style.  University  of 
Chicago  Press. 


4-24 


Documentation 


Annex  A:  Marking  Cover  Pages 


Distribution  Statements 

All  MCOTEA  T&E  documents  are 
marked  with  an  appropriate  distribution 
statement  (DOD  1987).  The  reference 
provides  policies  and  procedures  for  mark¬ 
ing  technical  data  for  release  and  dissemi¬ 
nation  without  additional  approvals  or  au¬ 
thorizations.  If  applicable,  all  MCOTEA 
T&E  documents  are  also  marked  with 
an  appropriate  export  control  warning 
in  accordance  with  DODD  5230.25. 

No  MCOTEA  document  is  distributed 
without  first  undergoing  a  proper  security 
classification  review  and  assignment  of  a 
distribution  statement. 

The  Division  Head  or  appropriate  staff  lead 
is  responsible  for  determining  the  distribu¬ 
tion  code  and  applicability  of  an  export 
control  warning  for  all  programs  assigned. 

Method 

MCOTEA  uses  the  guidance  contained  in 
this  section  to  select  an  appropriate  distribu¬ 
tion  statement.  Contractor  Sensitive  docu¬ 
ments  are  always  either  B  or  E. 

Proper  Marking  of  Documents 

The  document  originator  is  responsible 
for  ensuring  that  the  appropriate  mark¬ 
ings  are  applied.  The  distribution  statement 
is  displayed  conspicuously  on  electronic 
documents.  For  standard  printed  material, 
the  distribution  statement  appears  on  the 
front  cover  and  title  page,  if  any.  See  figure 
1  for  an  example. 

If  the  document  does  not  have  a  cover  or 
title  page  (such  as  forms,  spreadsheets, 
and  charts),  the  distribution  statement  is 
stamped,  printed,  written,  or  affixed  by 
other  means  in  a  conspicuous  position. 

Defense  Technical  Information 
Center 

All  reports  for  MCOTEA-led  assessments 
and  operational  evaluations  are  submit¬ 
ted  to  the  Defense  Technical  Information 
Center  (DTIC): 


♦  System  Assessment  Report 

♦  Intermediate  Assessment  Report 

♦  OTA  Assessment  Report 

♦  OTA  Milestone  Assessment  Report 

♦  OTA  Evaluation  Report 

♦  Follow-on  Evaluation  Report 

♦  Multi-Service  Evaluation  Report 

DTIC  Submission  Process 

MCOTEA  submits  reports  electronically  in 
.pdf  as  part  of  the  Program  Archive  Process. 
The  responsible  Division  or  staff  section  pre¬ 
pares  a  .pdf  copy  of  the  report  with  all  proper 
markings  on  the  cover  page  and  submits  that 
and  a  .pdf  copy  of  the  SF298  (Submission 
Form)  (fig.  3-4)  to  S-l.The  S-l  is  responsible 
for  establishing  and  maintaining  a  DTIC 
account  and  electronically  submitting  each 
report  via  the  DTIC  website. 

Cover  Page 

MCOTEA  documents  should  have  a  com¬ 
pleted  cover  page  that  includes  all  necessary 
information  identified  in  the  examples  shown 
in  figures  A-l  to  A-4. 

Proper  Marking  of  Documents 

The  document  originator  is  responsible  for 
ensuring  that  the  appropriate  markings  are 
applied.  The  distribution  statement  must 
be  displayed  conspicuously  on  electronic 
documents.  For  standard  written  or  printed 
material,  the  distribution  statement  appears 
on  each  front  cover  and  title  page.  See  the 
examples  in  figures  A-2  to  A-4.  The  majority 
of  MCOTEAs  documents  will  use  Distribu¬ 
tion  Statements  C  or  D. 

If  the  technical  information  is  not  prepared 
in  the  form  of  an  ordinary  document  and 
does  not  have  a  cover  or  title  page  (such  as 
forms,  spreadsheets,  and  charts),  the  appli¬ 
cable  distribution  statement  shall  be  stamped, 
printed,  written  or  affixed  by  other  means  in  a 
conspicuous  position. 


Definition  of 
Technical  Data 

Recorded  information 
related  to  experimental,  de¬ 
velopmental,  or  engineer¬ 
ing  works  that  can  be  used 
to  define  an  engineering 
or  manufacturing  process 
or  to  design,  procure, 
produce,  support,  maintain, 
operate,  repair,  or  overhaul 
material.  The  data  may  be 
graphics  and  pictures,  text 
in  specifications  or  related 
performance  or  design  type 
documents,  or  computer 
printouts.  Examples  of  tech¬ 
nical  data  include  research 
and  engineering  data, 
engineering  drawings,  and 
associated  lists,  specifica¬ 
tions,  standards,  process 
sheets,  manuals,  techni¬ 
cal  reports,  catalog-item 
identifications,  and  related 
information  and  computer 
software  documentation. 
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DISTRIBUTION  A.  Approved  for  public  release;  distribution  is  unlimited.  (*Note  - 
Documents  recommended  for  Public  Release  must  first  be  reviewed  in  accordance  with 
DoD  Directive  5230.9) 


DISTRIBUTION  B.  Distribution  authorized  to  U.S.  Government  Agencies  only  (fill 
in  reason)  (date  of  determination).  Other  requests  for  this  document  shall  be  referred  to 
Marine  Corps  Operational  Test  and  Evaluation  Activity,  2032  Barnett  Ave,  Quantico,  VA 
22134-5014. 


DISTRIBUTION  C.  Distribution  authorized  to  U.S.  Government  Agencies  and  their 
contractors  (fill  in  reason)  (date  of  determination).  Other  requests  for  this  document  shall 
be  referred  to  Marine  Corps  Operational  Test  and  Evaluation  Activity,  2032  Barnett  Ave, 
Quantico,  VA  22134-5014. 


DISTRIBUTION  D.  Distribution  authorized  to  DoD  and  U.S.  DoD  contractors  only 
(fill  in  reason)  (date  of  determination).  Other  requests  for  this  document  shall  be  referred  to 
Marine  Corps  Operational  Test  and  Evaluation  Activity,  2032  Barnett  Ave,  Quantico,  VA 
22134-5014. 


DISTRIBUTION  E.  Distribution  authorized  to  DoD  Components  only  (fill  in  reason) 
(date  of  determination).  Other  requests  for  this  document  shall  be  referred  to  Marine  Corps 
Operational  Test  and  Evaluation  Activity,  2032  Barnett  Ave,  Quantico,  VA  22134-5014. 


DISTRIBUTION  F.  Further  dissemination  only  as  directed  by  Marine  Corps  Opera¬ 
tional  Test  and  Evaluation  Activity,  2032  Barnett  Ave,  Quantico,  VA  22134-5014. 


DISTRIBUTION  X.  Distribution  authorized  to  U.S.  Government  Agencies  and  private 
individuals  or  enterprises  eligible  to  obtain  export-controlled  technical  data  in  accordance 
with  DoD  5230.25  (date  of  determination).  Controlling  DoD  office  is  Marine  Corps  Op¬ 
erational  Test  and  Evaluation  Activity,  2032  Barnett  Ave,  Quantico,  VA  22134-5014. 

EXPORT  CONTROL  WARNING  AND  DESTRUCTION  NOTICE.  In  addition 
to  the  Distribution  Statement  verbiage,  the  following  Export  Control  Warning  and  Destruc¬ 
tion  Notice  verbiage  must  also  be  listed  if  the  document  contains  technical  data  that  is  export 
controlled  (or  if  documentation  is  not  available  stating  otherwise): 

DESTRUCTION  NOTICE  -  For  classified  documents,  follow  the  procedures  in  DoD 
5220.22-M,  Industrial  Security  Manual,  Section  11  -19  or  DoD  5200.1-R,  Information 
Security  Program  Regulation,  Chapter  IX.  For  unclassified,  limited  documents,  destroy  by 
any  method  that  will  prevent  disclosure  of  contents  or  reconstruction  of  the  document. 

WARNING  -  This  document  contains  technical  data  whose  export  is  restricted  by  the 
Arms  Export  Control  Act  (Title  22,  U.S.C.,  Sec  2751,  et  seq.)  or  the  Export  Administration 
Act  of  1979,  as  amended, Title  50,  U.S.C.,  App.  2401  et  seq.  Violations  of  these  export  laws 
are  subject  to  severe  criminal  penalties.  Disseminate  in  accordance  with  provisions  of  DoD 
Directive  5230.25. 

Figure  2  is  an  example  SF298.  This  must  be  submitted  to  DTIC  with  the  report.  Figure  3  is 
the  Notice  to  Accompany  the  Dissemination  of  Export-Controlled  Technical  Data.  This  form 
is  enclosure  5  to  DOD  5230.25  and  must  accompany  any  reprinted,  export-controlled  techical 
documents. 
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[System  Name] 

[Type  of  Event]  Evaluation  Report 


(Date) 

Marine  Corps  Operational  Test  and  Evaluation  Activity 
2032  Barnett  Avenue 
Quantico,  VA  22134-5014 


Approved: 

[Director’s  Name] 

Colonel  USMC 

Director,  MCOTEA 

Date 

DISTRIBUTION  C.  Distribution  authorized  to  U.S.  Government  Agencies  and  their  contractors 
(fill  in  reason)  (date  of  determination).  Other  requests  for  this  document  shall  be  referred  to  Marine 
Corps  Operational  Test  and  Evaluation  Activity,  2032  Barnett  Ave,  Quantico,  VA  22134-5014. 

WARNING  -  This  document  contains  technical  data  whose  export  is  restricted  by  the  Arms  Export 
Control  Act  (Title  22,  U.S.C.,  Sec  2751,  et  seq.)  or  the  Export  Administration  Act  of  1979,  as 
amended,  Title  50,  U.S.C.,  App.  2401  et  seq.  Violations  of  these  export  laws  are  subject  to  severe 
criminal  penalties.  Disseminate  in  accordance  with  provisions  of  DoD  Directive  5230.25. 

DESTRUCTION  NOTICE  -  For  classified  documents,  follow  the  procedures  in  DoD  5220.22-M, 
Industrial  Security  Manual,  Section  11  -19  or  DoD  5200. 1-R,  Information  Security  Program 
Regulation,  Chapter  IX.  For  unclassified,  limited  documents,  destroy  by  any  method  that  will 
prevent  disclosure  of  contents  or  reconstruction  of  the  document. 


1.  C  is  appropriate  choice  because  plans  are  not  contractor  sensitive  and  rea¬ 
son  does  not  exist  to  limit  distribution  to  DOD  and  their  Contractors  only. 
If  this  were  an  Evaluation  Report,  C  would  still  be  appropriate  in  this  case 
because  the  CAC2S  system  is  post  source  selection. 

2.  Critical  Technology  is  appropriate  because  the  System  Evaluation  Plan 
contains  information  that  could  be  used  to  identify  system  capabilities  or 
limitations. 

3.  The  Export  Control  Warning  and  Destruction  Notice  verbiage  must  be 
listed  if  the  document  contains  technical  data  that  is  export  controlled  (or  if 
documentation  is  not  available  stating  otherwise) 


Figure  1. 

Complete  cover  template 
with  all  possible  security 
markings. 
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REPORT  DOCUMENTATION  PAGE 

Form  Approved 

OMB  No.  0704  0188 

The  public  reporting  burden  for  this  collection  of  information  is  estimated  to  average  1  hour  per  response,  including  the  time  for  reviewing  instructions,  searching  existing  data  sources, 
gathering  and  maintaining  the  data  needed,  and  completing  and  reviewing  the  collection  of  information.  Send  comments  regarding  this  burden  estimate  or  any  other  aspect  of  this  collection 
of  Information,  including  suggestions  for  reducing  the  burden,  to  Department  of  Defense.  Washington  Headquarters  Servoes.  Directorate  for  Information  Operations  and  Reports 
(0704  0188).  1215  Jefferson  Davis  Highway.  Suite  1204.  Arlington.  VA  22202-4302  Respondents  should  be  awaie  (hot  notwithstanding  any  other  provision  of  law,  no  person  shall  be 
subject  to  any  penalty  for  falling  to  comply  with  a  collection  of  Infoimotion  if  it  does  not  display  a  currently  valid  OMB  conirol  number 

PLEASE  DO  NOT  RETURN  YOUR  FORM  TO  THE  ABOVE  ADDRESS. 

1.  REPORT  DATE  (DO  MM  YYYY,  2.  REPORT  TYPE 

xx-xx-xxxx  [System  Name]  Evaluation  Report 

3.  DATES  COVERED  (From  -  To) 

xx/xx/  xxx-xx/xx/  xxxx 

4.  TITLE  AND  SUBTITLE 

[System  Name,  Increment/Phase] 

5a.  CONTRACT  NUMBER 

5b.  GRANT  NUMBER 

5c.  PROGRAM  ELEMENT  NUMBER 

6.  AUTHOR1S! 

Name  of  Originator  at  MCOTEA 

5d.  PROJECT  NUMBER 

5e.  TASK  NUMBER 

5f.  WORK  UNIT  NUMBER 

7.  PERFORMING  ORGANIZATION  NAME(S)  AND  ADDRESSES! 

Marine  Corps  Test  and  Evaluation  Activity 

2032  Barnett  Avenue 

Quantico,  VA  22134-5014 

8.  PERFORMING  ORGANIZATION 

REPORT  NUMBER 

9.  SPONSORING/MONITORING  AGENCY  NAMEIS)  AND  ADDRESS1ES1 

10.  SPONSOR/MONITOR  S  ACRONYMIS] 

MCOTEA 

11.  SPONSOR/MONITOR'S  REPORT 
NUMBER(S) 

12.  DISTRIBUTION/AVAILABILITY  STATEMENT 

Distribution  Code  C  -  Distribution  limited  to  U.S.  Government  Agencies  only:  Critical  Technology;  [Month/Year].  Refer  other 
requests  fpr  this  document  to  the  Marine  Corps  Operational  Test  and  Evaluation  Activity,  2023  Barnett  Avenue 
Quantico,  VA  2213-5014 

13.  SUPPLEMENTARY  NOTES 


14.  ABSTRACT 

[Single-sentence  description  of  document] 


15.  SUBJECT  TERMS 

[Keywords.  Example:  Command  and  Control,  Aviation] 


16.  SECURITY  CLASSIFICATION  OF: 

17.  LIMITATION  OF 
ABSTRACT 

18  NUMBER 
OF 

PAGES 

[xxx] 

19a.  NAME  OF  RESPONSIBLE  PERSON 

a.  REPORT 

b.  ABSTRACT 

c.  THIS  PAGE 

Scientific  Advisor 

Unclassified 

Unclassified 

Unclassified 

uu 

19b,  TELEPHONE  NUMBER  (Include  urea  code, 

Standard  Form  298  (Rev.  8/981 


Figure  4. 

Example  of  a  completed  SF  298. 
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NOTICE  TO  ACCOMPANY  THE  DISSEMINATION  OF  EXPORT-CONTROLLED 

TECHNICAL  DATA 


E5. 1 . 1 .  Export  of  information  contained  herein,  which  includes,  in  some 
circumstances,  release  to  foreign  nationals  within  the  United  States,  without  first 
obtaining  approval  or  license  from  the  Department  of  State  for  items  controlled  by  the 
International  Traffic  in  Arms  Regulations  (ITAR),  or  the  Department  of  Commerce  for 
items  controlled  by  the  Export  Administration  Regulations  (EAR),  may  constitute  a 
violation  of  law. 

E5. 1 .2.  Under  22  U.S.C.  2778  the  penalty  for  unlawful  export  of  items  or 
information  controlled  under  the  ITAR  is  up  to  2  years  imprisonment,  or  a  fine  of 
SI 00,000,  or  both.  Under  50  U.S.C.,  Appendix  2410,  the  penalty  for  unlawful  export 
of  items  or  information  controlled  under  the  EAR  is  a  fine  of  up  to  $  1 ,000,000,  or  five 
times  the  value  of  the  exports,  whichever  is  greater;  or  for  an  individual,  imprisonment 
of  up  to  1 0  years,  or  a  fine  of  up  to  $250,000,  or  both. 

E5.1.3.  In  accordance  with  your  certification  that  establishes  you  as  a  "qualified 
U.S.  contractor,"  unauthorized  dissemination  of  this  information  is  prohibited  and  may 
result  in  disqualification  as  a  qualified  U.S.  contractor,  and  may  be  considered  in 
determining  your  eligibility  for  future  contracts  with  the  Department  of  Defense. 

E5.1.4.  The  U.S.  Government  assumes  no  liability  for  direct  patent  infringement, 
or  contributory  patent  infringement  or  misuse  of  technical  data. 

E5.1 .5.  The  U.S.  Government  does  not  warrant  the  adequacy,  accuracy,  currency, 
or  completeness  of  the  technical  data. 

E5.1.6.  The  U.S.  Government  assumes  no  liability  for  loss,  damage,  or  injury 
resulting  from  manufacture  or  use  for  any  purpose  of  any  product,  article,  system,  or 
material  involving  reliance  upon  any  or  all  technical  data  furnished  in  response  to  the 
request  for  technical  data. 

E5.1.7.  If  the  technical  data  furnished  by  the  Government  will  be  used  for 
commercial  manufacturing  or  other  profit  potential,  a  license  for  such  use  may  be 
necessary.  Any  payments  made  in  support  of  the  request  for  data  do  not  include  or 
involve  any  license  rights. 

E5. 1 .8.  A  copy  of  this  notice  shall  be  provided  with  any  partial  or  complete 
reproduction  of  these  data  that  are  provided  to  qualified  U.S.  contractors. 


Figure  2. 

Notice  to  Accompany  the  Dissemination  of 
Export-Controlled  Technical  Data. 
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Asking  the  MDA 

Was  MCOTEA  able  to  help  you 
identify  system  risk  areas  early 
in  the  development  cycle? 

What  parts  of  MCOTEA's  efforts 
do  you  find  most  useful?  Least 
useful? 

Was  I0T/F0T  conducted  fairly? 
Realistically? 

Was  the  evaluation  of  the 
system  fair  and  accurate? 

Were  the  items  identified 
by  MCOTEA  as  Major  System 
Deficiencies  reasonable? 

Were  the  items  identified 
by  MCOTEA  as  Operational 
Deficiencies  reasonable? 

How  can  MCOTEA  better  help 
the  acquisition  process  in  the 
future? 

Do  you  have  any  suggestions  to 
improve  MCOTEA  processes? 


Process  Feedback 

MCOTEA  continuously  strives  to  improve 
its  processes  to  ensure  that  MCOTEA’s 
assessments,  tests,  and  evaluations  are 
relevant,  timely,  accurate,  unbiased,  and 
operationally  useful.  Therefore,  MCOTEA 
solicits  feedback  from  diverse  sources  to 
improve  existing  processes  and  to  identify 
the  need  for  potential  new  processes.  Any 
suggestions  for  potential  improvements  to 
MCOTEA  processes  should  be  forwarded 
to  the  Scientific  Advisor  for  consideration. 

PM  and  MDA  Feedback 

Both  the  PM  and  the  MDA  are  focused  on 
delivering  a  relevant  warfighting  capability 
on  schedule  and  within  cost.  Although 
MCOTEA  OT  might  be  viewed  as  a 
type  of  program  “final  exam, ’’MCOTEA’s 
involvement  in  a  program  helps  the  PM  and 
MDA  to  lower  risk  and  deliver  the  needed 
operational  capability  to  the  Warfighter. 

Within  30  days  of  sending  the  final  SAR 
or  OER  to  the  PM  and  MDA,  the  test 
team  solicits  feedback  from  them.  The 
test  team  interviews  the  PM  (telephonic 
interview  suffices)  using,  at  a  minimum, 
the  initial  set  of  high-level  interview 
questions  shown  in  the  sidebar.  These 
questions  are  intended  as  a  starting  point. 
The  interviewer  is  expected  to 
pose  each  question  and  pursue 
each  line  of  questioning  with 
an  eye  toward  identifying 
how  MCOTEA  can  better 
perform  the  next  system 
evaluation  or  assessment.  In 
addition  the  test  team  sends  a 
survey  to  the  MDA.  The  test 
team  summarizes  the  results  of 
the  interview  and  the  survey 
and  forward  them  to  the 
Scientific  Advisor  for  analysis 
and  archiving.  The  Scientific 
Advisor  may  immediately 
recommend  MCOTEA 


process  changes  or  may  recommend 
changes  after  accumulating  evidence  of  a 
needed  change  over  time. 

Operational  Advisory  Group 
Feedback 

MCOTEA  attends  Operational  Advisory 
Groups  (OAG),  which  provide  MCOTEA 
with  the  latest  operational  feedback  on 
existing  weapon  systems,  short  of  going  to 
the  field.  To  the  extent  that  these  groups 
discuss  systems  that  have  been  through  the 
MCOTEA’s  test  and  evaluation  process, 
these  forums  may  present  an  opportunity 
for  MCOTEA  to  discover  potential  areas 
of  provement  in  its  processes.  MCOTEA’s 
OAG  attendance  occurs  typically  on  an  ad 
hoc  basis.  Any  member  of  the  MCOTEA 
staff  attending  an  OAG  should  be  alert  for 
potentially  useful  changes  to  MCOTEA 
processes  and  forward  a  written  summary 
and  justification  for  potential  changes  to 
the  Scientific  Advisor. 

Warfighter  Feedback 

The  best  way  to  obtain  operational 
information  on  specific  systems  is  to 
conduct  structured  interviews  (see  sidebar) 
with  Warfighters  who  are  using  these 
systems.  The  idea  is  to  determine  areas  of 


Asking  the  PM 

Was  MCOTEA  able  to  help  you  identify  system  risk  areas  early  in  the 
development  cycle? 

How  can  MCOTEA  participation  in  DT  events  be  improved? 

Was  the  evaluation  of  the  system  fair  and  accurate? 

Were  the  items  identified  by  MCOTEA  as  Major  System  Deficiencies 
reasonable? 

Were  the  items  identified  by  MCOTEA  as  Operational  Deficiencies 
reasonable? 

How  can  MCOTEA  better  help  the  acquisition  process  in  the  future? 
Do  you  have  any  suggestions  to  improve  MCOTEA  processes? 
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deficiency  that  may  or  may  not  have  been 
identified  during  MCOTEA  OT&E. 

Purpose  of  Warfighter  Feedback 

Obtaining  Warfighter  feedback  is 
a  valuable  exercise  designed  to  help 
all  phases  of  the  acquisition  process 
deliver  the  capabilities  most  needed  by 
Marines.  Warfighter  feedback  is  useful 
to  the  materiel  developer  and  DC, 

CD&I  (USMC  2010).  MCOTEA 
uses  Warfighter  feedback  to  examine 
the  effectiveness  of  its  processes  and 
to  improve  them  as  necessary.  If  the 
Warfighters  in  the  field  and  MCOTEA 
identify  the  same  system  deficiency,  then 
no  process  improvements  are  indicated. 
However,  if  for  some  reason  MCOTEA’s 
operational  test  and  evaluation  missed  a 
deficient  area  identified  by  Warfighters,  a 
MCOTEA  process  or  procedure  might 
need  improvement. 

Selecting  Systems  for 
Warfighter  Feedback 

The  best  candidates  for  feedback  are 
systems  that  have  been  deployed  for  a 
long  enough  period  of  time  to  allow 
operators  to  become  familiar  with  them, 
but  are  recent  enough  so  the  results  of 
the  feedback  will  affect  current  processes. 
Therefore,  eligibilty  as  a  system  of  interest 
requires  the  system  to  be  at  least  1  year 
past  its  initial  deployment  date,  but  not 
more  than  2  years  past  initial  deployment. 
At  least  two  different  systems  (from  two 
different  product  groups  or  programs)  will 
be  examined  for  feedback  each  year. 

The  COT,  S-2,  Scientific  Advisor,  and 
representatives  from  MCSC/PEO-LS  and 
the  appropriate  Integration  Divisions  within 
the  Capabilities  Development  Directorate 
of  CD&I  compose  the  Warfighter 
Feedback  IPT,  which  selects  systems  for 
feedback.  The  MCOTEA  COT  chairs  the 
Warfighter  Feedback  IPT,  which  meets  in 
the  third  quarter  of  each  fiscal  year  to  select 
the  systems  of  interest  for  the  next  fiscal 
year.  The  lead  members  of  the  field  team 


(one  each  from  DC,  CD&I;  the  materiel 
developer;  and  MCOTEA)  are  designated 
at  the  IPT  meeting. 

Obtaining  Warfighter  Feedback 

Feedback  is  obtained  after  the  start  of 
the  fiscal  year,  but  before  the  Warfighter 
Feedback  IPT  meets  in  the  third  quarter 
of  the  fiscal  year  to  select  the  systems 
of  interest  for  the  next  fiscal  year.  This 
gives  the  Warfighter  Feedback  IPT  the 
benefit  of  the  most  recent  feedback  before 
selecting  the  next  year’s  systems  of  interest. 

A  team  consisting  of  at  least  one  individual 
from  DC,  CD&I;  MCOTEA;  and  the 
materiel  developer  uses  a  structured 
interview  process  to  obtain  feedback  on  the 
system  of  interest  from  a  variety  of  system 
users  at  various  levels  in  the  command 
structure.  MCOTEA  members  of  the 
team  use,  at  a  minimum,  the  initial  set  of 
high-level  interview  questions  as  shown  on 
page  5-4.  These  questions  are  intended  as  a 
starting  point.  The  interviewer  is  expected 
to  pose  each  question  and  pursue  each 
line  of  questioning,  identifying  areas  of 
improvement  in  MCOTEA  processes.  The 
interviewers  should  be  aware  that  deployed 
operators  have  other  tasks  for  which  they  are 
responsible.  Data  on  the  system  of  interest 
will  be  gathered  in  a  way  that  does  not 
interfere  with  the  Warfighter’s  primary  job. 

Documenting  Warfighter  Feedback 

The  MCOTEA  lead  team  member  ensures 
that  the  results  of  the  structured  interview 
are  documented  by  summarizing  the  results 
of  each  high-level  interview  question.  This 
summary  is  forwarded  to  the  Scientific 
Advisor  for  analysis  and  archiving.  The 
report  is  finished  within  30  days  of  the 
interview  team’s  return  from  the  field. 

MCOTEA  Test  Team  and 
Evaluation  Team  Feedback 

MCOTEA’s  test  teams  comprise  a  rich 
potential  source  of  constructive  feedback 
regarding  MCOTEA  processes.  After  each 


A  structured  interview 
starts  with  a  set  of 
general  questions 
covering  a  wide 
range  of  topics  and 
is  intended  to  allow 
the  interviewer 
to  ask  follow-on 
questions  based  on 
the  interviewee's 
responses.  The 
interviewer  is 
expected  to  "drill 
down" on  each 
question  to  discover 
the  important  issues 
and  obtain  the 
required  information. 
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operational  test  and  before  the  participating 
Marines  return  to  their  units,  the  test  team 
meets  with  the  participating  Marines 
from  the  Operating  Forces,  including  the 
test  MOIC  or  NCOIC,  to  conduct  a  hot 
wash.  This  meeting  ascertains  the  lessons 
learned  during  the  operational  test  and 
suggest  improvements  for  MCOTEA’s 
test  processes.  During 
the  hot  wash,  the  test 
process  should  be  broken 
into  parts  corresponding 
to  the  sections  in  chapter 
3  of  this  Manual  to 
facilitate  drawing  out 
any  suggestions.  In 
addition,  within  30 
days  after  each  system 
evaluation  is  concluded, 
the  OA  conducts  a 
hot  wash  with  the  test 
team  to  determine 
lessons  learned  and 
suggest  evaluation 
process  improvements. 

The  lessons  learned  are 
recorded  as  described 
later  in  this  chapter. 

Examining 
Existing 
Databases 

With  the  appropriate 
approval,  COT  may  probe  existing 
database  information  from  time  to  time 
on  deployed  systems  to  help  identify 
potential  process  improvements.  Deficiency 
reports  are  the  type  of  information 
typically  available  and  can  indicate  how 
specific  systems  are  doing  in  the  field. 
Although  these  databases  do  not  normally 
contain  enough  information  to  calculate 
some  suitability  terms,  such  as  reliability 
or  availability,  they  can  be  useful  in 
determining  others,  such  as  transportability 
and  safety.  If  these  databases  contain  data 
conflicting  with  MCOTEA  calculations 
used  to  determine  Operational  Suitability, 
the  need  to  re-examine  the  relevant 


MCOTEA  process  may  be  indicated. 

An  example  of  a  potentially  useful  database 
is  the  Product  Data  Reporting  and 
Evaluation  Program  run  by  the  Marine 
Corps  Logistics  Command  in  Albany,  GA, 
which  manages  Product  Quality  Deficiency 
Reports  (PQDR).The  PQDR  reports 

deficiencies  occurring  in 
major  weapon  systems, 
secondary/ consumable/ 
repairable  items, 
spare  and  repair  parts, 
government-owned 
products  used  during 
development  and  test, 
and  items  supplied  as 
Government- Furnished 
Property,  to  include 
warranted,  contractor 
logistics  support, 
commercial-off-the- 
shelf,  and  Marine  Corps 
common  hardware 
suite  items.  Analyzing 
PQDRs  on  selected 
systems  will  be  most 
helpful  in  examining  the 
MCOTEA  process  for 
determining  OS. 

Lessons 
Learned 

Recording  MCOTEA  Lessons 
Learned 

Lessons  learned  may  be  MCOTEA’s  most 
underestimated  resource.  Through  lessons 
learned  a  future  test  or  evaluation  team  can 
discover  ways  to  streamline  processes  and 
avoid  errors  and  frustrations. 

Lessons  are  learned  throughout  the 
MCOTEA  process.  It  is  important  to 
record  those  lessons  soon  after  they  are 
recognized  as  being  potentially  useful  to 
future  MCOTEA  analyses,  interactions, 
and  operations.  Categories  and  key  words 
are  used  to  facilitate  future  access. 

Any  member  of  MCOTEA  (military, 


Asking  the  Warfighter 

How  long  have  you  been  using  the 
system? 

Under  what  circumstances  have  you  used 
the  system? 

How  would  you  improve  on  the 
effectiveness  of  the  system? 

Is  the  system  easy  to  use,  transport, 
maintain? 

Is  the  system  compatible  and 
interoperable  with  the  other  systems  you 
use? 

Are  there  any  safety  issues  with  the 
system? 

Is  the  system  easily  supportable? 

How  often  does  the  system  fail,  and  how 
does  it  fail? 

Are  their  any  tactical  disadvantages  to 
using  the  system? 

Does  the  system  impact  your  personal 
survivability?  How? 
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government  civilian,  or  contractor)  can 
contribute  to  the  lessons  learned  effort. 
The  OTPO  is  responsible  for  collecting 
and  submitting  lessons  learned  during  the 
test  process.  The  OA  is  responsible  for 
collecting  and  submitting  lessons  learned 
during  the  evaluation  process. 

All  stages  of  the  test  and  evaluation 
process  from  program  definition  through 
archiving  are  subject  to  lessons  learned. 
Lessons  learned  should  also  be  submitted 
for  any  MCOTEA  function  where  the 
observations  and  recommendations  may 
lead  to  improved  organizational  efficiency 
or  effectiveness. 


30  days  of  final  report  delivery. 

Once  a  lesson  learned  is  entered,  it  is 
placed  in  “pending”  status.  While  in  this 
status,  the  lesson  learned  is  available  only 
to  users  registered  under  MCOTEA  as 
their  Major  Command.  Once  a  month,  the 
Chief  of  Test  examines  “pending”  lessons 
learned  and  determines  which  of  them  can 
be  released  from  that  status.  Once  released, 
lessons  learned  are  visible  to  any  authorized 
user  of  the  MCCLL  database.  Only  the 
COT  may  release  MCOTEA’s  lessons 
learned  from  “pending”  status. 

Using  the  MCCLL  Database 


Lessons  learned  are  submitted  using  the 
Marine  Corps  Center  for  Lessons  Learned 
(MCCLL)  website  at  https://www.mcdl. 
usmc.mil  (fig.  5-1).  The  site  is  located  on 
NIPRNET  and  is  Common  Access  Card 
(CAC)-enabled.  Using  this  website  aUows 
a  MCOTEA  member  to  enter  lessons 
learned  from  any  site  with  Internet  access 
while  the  lesson  remains  fresh.  Entering 
lessons  learned  into  the  MCCLL  database 
does  not  replace  the  requirement  to  forward 
a  complete  package  of  all  lessons  learned 
during  a  program.  Test  lessons  learned  are 
submitted  to  the  Chief  of  Test  (via  the 
Division  Head),  and  evaluation  lessons 
learned  are  submitted  to  the  COT  via  the 
OA  for  assessments  and  evaluations  within 


First-time  users  must  establish  an  account 
with  MCCLL  and  specify  that  they  are 
MCOTEA  users  in  the  Major  Command 
menu.  (The  site  can  only  be  accessed  via 
CAC.)  After  logging  on,  users  can  add 
observations  or  recommendations  by 
clicking  the  action  menu.  The  next  screen 
(fig.  5-2)  shows  a  series  of  pull-down  menus 
that  determine  the  appropriate  category 
for  entering  information.  Only  menus 
beginning  with  “MCOTEA”  are  used. 

“Overall  Classification”  is  always 
marked  “unclassified.  ”  “Record  Type” 
is  always  marked  “Observation  and 
Recommendation.”  “Operational”  is  always 
marked  “N/A.  ”  “MCOTEA”  is  always 
indicated  as  the  Major  Command. 


MiltiWMfflffHTtlff.B-imH 
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SAFETY  CORNER 
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the  Action 
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search 

Dservations  and 
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Afghanistan  Information  Center  -  ^AST  L''p~A  TED  130121  Read  more. 


Haiti  Humanitarian  Assistance/Disaster  Relief  Read  more. 
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Figure  5-1. 
MCCLL  Home 
Page 
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Figure  5-2. 
MCOTEA-specific 
Categories 


Only  information  pertaining  to  the 
particular  lesson  is  added  under  the 
“Campaign/Operation/Exercise” 
options.  After  entering  the  appropriate 
information  in  pull-down  menus,  users 
can  enter  the  relevant  information  in  the 
fields  labeled  “Topic/Issue.”  Additionally, 
the  program  name  can  be  entered  in 
this  field,  if  applicable,  or  Observation, 
Discussion,  Recommendation, 
Implications,  and  Event  Description. 

To  search  MCCLL  for  observations  or 
recommendations,  users  click  the  action 
menu  from  the  home  page  and  choose 
from  any  combination  of  the  available 
menus.  Registered  MCOTEA  users  can 
select  and  read  MCOTEA  lessons  learned, 
even  if  they  are  still  in  “pending”  status. 

Archiving  and  the  T&E 
Reference  Center 

Any  document  signed  by  the  Director  or 
by  direction  of  the  Director  or  a  Division 
Head  is  an  official  record  that  MCOTEA 
must  maintain. 

In  addition,  MCOTEA  maintains  a  Test 
and  Evaluation  Reference  Center  (TERC) 
of  the  program  documents  and  data 


obtained  during  system  test  and  evaluation, 
located  on  MCEITS.  Data  stored  in  the 
TERC  is  associated  with  the  documented 
test  setup,  the  conditions  under  which  the 
data  was  gathered,  and  the  evaluation.  The 
data  is  stored  in  a  way  that  maintains  its 
integrity,  in  conjunction  with  the  idea  that 
the  data  might  be  reanalyzed  in  the  future. 

The  TERC  comprises  three  functional 
parts:  a  data  repository,  an  active  program 
working  area,  and  a  permanent  storage  area 
as  shown  in  figure  5-3. 

The  TERC  is  intended  to  house  supporting 
information  for  all  current  and  past 
MCOTEA  programs  over  the  previous 
20  years.  In  that  sense  it  is  a  working 
knowledge  center  much  like  an  electronic 
library.  The  data  repository  serves  as  the 
place  where  test  data  is  stored  during 
testing  and  while  it  is  being  analyzed. 

Once  the  data  has  served  its  use  in  the 
MCOTEA  evaluation,  it  is  moved  to  the 
permanent  storage  area  of  the  TERC. 

The  active  program  working  area  is  where 
the  documents  under  construction  reside. 
Once  a  document  attains  final  signature,  or 
is  otherwise  completed,  it  too  is  moved  to 
the  permanent  storage  area  of  the  TERC. 
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Figure  5-3.  Components  of  the  TERC 

The  data  in  the  TERC  may  be  used  for 
establishing  “current”  system  capability 
when  MCOTEA  evaluates  the  future 
replacement  for  the  current  system.  This 
“current”  capability  can  be  used  to  compare 
with  the  future  system.  The  “current” 
capability  can  also  be  used  to  establish  the 
corresponding  data  point  when  executing 
the  MCOTEA  process  in  evaluating 
Mission  Capability  Level.  Given  the  time 
required  to  establish  the  need  to  replace  a 
current  system  and  get  the  new  system  to 
operational  testing,  MCOTEA  maintains 
all  program-related  documents  and 
data  files  in  the  TERC  for  20  years.  All 
programs,  regardless  of  ACAT  designation, 
that  are  examined  in  some  way  by 
MCOTEA,  will  have  their  own  program 
files  in  the  MCOTEA  T&E  Reference 
Center;  this  includes  AAP,  ACAT  IV(M), 
and  QRA  programs. 

This  section  contains  the  types  of  official 
(and  certain  unofficial)  records  maintained 
at  MCOTEA,  as  well  as  instructions 
for  the  proper  storage  and  disposition  of 
records.  The  S-l  forwards  recommended 
changes  to  the  types  of  records  maintained 
by  MCOTEA  to  the  Director  for  approval. 

Standard  Subject  Identification 
Code  (SSIC) 

In  accordance  with  SECNAV  M-5210.2, 
chapter  2  (SECNAV  2008),  OT&E 
documents  are  assigned  the  SSIC  of 
YYYY/X/MCOTEA,  where  YYYY  is 
the  appropriate  standard  identifier  and  X 
is  a  number  assigned  to  the  originating 


MCOTEA  office,  Division,  or  Section 
(table  5-1).  The  standard  identifier  for 
“General  Test  and  Evaluation  Records” 
is  3980.  Appending  the  SSIC  with 
“MCOTEA”  helps  recover  information 
from  the  National  Archives  in  the  event  of 
inadvertent  loss  of  data  in  the  MCOTEA 
TERC.  At  MCOTEA,  the  SSIC  is 
normally  assigned  to  the  transmittal  letter 
assigned  to  each  document. 


Table  5-1.  MCOTEA  SSIC  codes 


MCOTEA  Office 

SSIC 

Director 

3980/01/MCOTEA 

Deputy  Director 

3980/02/MCOTEA 

Scientific  Advisor 

3980/03/MCOTEA 

Chief  of  Test 

3980/04/MCOTEA 

Lead  Contract  Integrator 

3980/05/MCOTEA 

Chief  of  Staff 

3980/06/MCOTEA 

Combat  Service  Support  Test  Division 

3980/07/MCOTEA 

Expeditionary  Test  Division 

3980/08/MCOTEA 

Ground  Combat  Test  Division 

3980/09/MCOTEA 

MAGTF/C4ISR  Test  Division 

3980/10/MCOTEA 

S-1 

3980/1 1/MCOTEA 

S-2 

3980/1 2/MC0TEA 

S-3 

3980/1 3/MC0TEA 

S-4 

3980/14/MCOTEA 

S-5 

3980/1 5/MC0TEA 

Fiscal 

3980/16/MCOTEA 

Staff  Responsibilities 

S- 1  Lead 

The  S-l  oversees  the  MCOTEA  TERC. 

At  a  minimum,  the  S-l  Lead 

♦  maintains  MCOTEA  program  documents 
and  data  files  and  non-MCOTEA 
program  documents  and  data  files  for  20 
years  in  accordance  with  the  guidance  in 
this  Manual 

♦  maintains  the  official  records  of  the  Activity 
in  accordance  with  the  guidance  in  this 
Manual  and  SECNAVINST  5210.1,  page 
III-3-60 

♦  conducts  an  annual  review  (usually  in 
January)  of  MCOTEA’s  records  in 
accordance  with  the  direction  in  this  section 


OTPO 

Responsibilities 

■=>  Provide  the  S-1  a 
copy  of  all  documents 
signed  by  the  Director 
as  part  of  the  routing 
process.  Documents 
signed  by  direction 
of  the  Director  via 
Division  Heads  and 
Section  Leads  are  also 
routed  to  the  S-1. 

■=>Use  the  MCOTEA 
T&E  Reference 
Center  for  storing 
and  maintaining  all 
program  documents 
and  data  files  while 
the  program  is  active 
and  after  it  closes 
to  ensure  that  these 
valuable  resources  are 
accessible  to  MCOTEA 
and  future  officers 
assigned  to  the 
program. 

■=>  Coordinate  with 
the  S-1  to  complete  a 
thorough  review  of  the 
OT&E  documentation 
and  data  files  related 
to  a  program  no  later 
than  45  days  after 
program  completion 
or  abandonment  to 
ensure  proper  filing 
in  the  T&E  Reference 
Center. 
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Division  Heads  and  S-2  Lead 

The  Division  Heads  and  the  S-2  Lead  are 
responsible  for  stressing  the  importance  of 
meticulous  record  keeping  and  archiving 
to  their  personnel.  Proper  record  keeping 
is  considered  a  key  element  of  successful 
OT&E. 

Operational  Test  Project  Officers 

The  most  important  records  maintained  by 
this  Activity  are  the  OT&E  documents  and 
data  files.  While  oversight  of  MCOTEA’s 
T&E  Reference  Center  is  the  responsibility 
of  the  S-l,  it  is  imperative  that  the  test  team 
be  involved  in  the  collection  and  storage  of 
program-related  records. 

These  files  include  program-specific 
documents  including  all  official  test 
planning,  evaluation,  and  reporting 
documents  and  all  data  files  generated 
during  test  and/or  evaluation.  After  program 
closure,  they  are  stored  in  the  MCOTEA 
T&E  Reference  Center.  Documents  and 
data  files  generated  by  outside  agencies, 
which  MCOTEA  uses  to  evaluate  the 
system,  are  also  stored  in  the  TERC. 

OT&E  Program  Documentation 
and  Data  Files 

OT&E  documentation  and  data  files 
maintained  by  the  S-l  will  be  in  electronic 
form  (for  ease  of  access)  to  the  maximum 
extent  possible.  Hard  copies  are  maintained 
when  electronic  conversion  is  not  possible. 
See  table  5-2  for  a  summary  of  the 
requirements  for  each  form  of  storage: 

♦  All  hard  copies,  CDs,  audio  tapes,  and 
video  tapes,  as  well  as  any  other  hard 
copy  program  documentation,  are  labeled 
with  the  program  name,  SSIC,  and  any 
additional  information  to  aid  future 
information  reference  or  recovery. 

♦  Per  SECNAVINST  5210.1,  copies  of 
program  documents  in  the  program 
folder  are  sent  to  the  National  Archives 
for  permanent  storage  3  years  after  the 
file  is  closed  ( Notes  to  Chapter  5  contains 


information  on  submitting  files  to  the 
National  Archive).  MCOTEA’s  S-l 
maintains  the  originals  for  20  years  after  the 
file  is  closed. 

♦  To  the  maximum  extent  possible,  the  S-l 
maintains  an  electronic  copy  of  all  OT &E 
documents  and  data  files.  Magnetic  tape 
and  audio  or  video  tapes  will  be  maintained 
without  alteration.  For  active  programs, 
the  documents  and  data  files  are  stored 

in  the  TERC  by  the  cognizant  Division. 
MCOTEA  maintains  all  data  files, 
magnetic  tape,  and  audio  or  video  tapes, 
closed  upon  completion  or  abandonment  of 
the  OT&E,  for  20  years. 

Other  Relevant  Program  Files 

OTPOs  are  responsible  for  maintaining 
OT&E  program-related  files  and 
documents  that  are  not  signed  by  the 
Director  or  by  direction  of  the  Director  or 
Division  Head,  but  are  considered  relevant 
records  of  the  programs  OT&E,  either 
electronically  on  disk  or  in  hard  copy  form. 
When  the  program  is  closed,  all  documents 
and  data  files  are  transferred  to  the  T&E 
Reference  Center  and  stored  under  the 
appropriate  metadata: 

♦  Program  Name 

♦  PEO  or  Systems  Command  Program  Office 

♦  ACAT  level,  AAP,  or  QRA 

♦  MCOTEA  Test  Division  and  Branch 

♦  Document  Type 

♦  Date  (month/year) 

♦  Signed?  Y/N 

Archiving  Program  Documents 
and  Data  Files 

An  OT&E  program  remains  open  until 
the  OTPO  has  signed  and  completed  a 
Program  Closure  Form,  an  example  of 
which  appears  in  figure  5-4.  Once  signed, 
the  OTPO  delivers  the  form  to  the  S-l. 
The  S-l  then  notifies  all  key  MCOTEA 
personnel  (all  personnel  with  a  MCOTEA 
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Table  5-2.  Summary  of  Document  Storage  and  Disposition 

Historic  OT&E  Policy  Correspondence  MOA/MOU 

Fi  loe  Hnrc  I  otforc  ' 


Assigned  an  SSIC 

y 

y 

OTPO  Responsibility 

y 

y 

S-1  Responsibility 

y 

y 

y 

y 

y 

Stored  for  20  Years  at  MCOTEA 

y 

Stored  Permanently  at  MCOTEA 

y 

y 

y 

Stored  for  2  Years  at  MCOTEA 

y 

SSIC)  that  the  program  has  been  closed. 
Once  notified,  the  test  team  and  anyone 
else  at  MCOTEA  with  relevant  program 
documentation,  information,  and  data  are 
directed  to  deliver  their  information  to  the 
S-l  for  T&E  Reference  Center  filing  within 
30  days.  The  OTPO  conducts  a  thorough 
audit  of  the  S-l’s  archive  no  later  than  45 
days  after  closure  to  ensure  that  all  relevant 
records  have  been  appropriately  filed. 


and  events  shown  in  table  5-4.  This 
information  is  tracked  by  the  OTPO  in 
the  Program  Resource  Tracking  Survey. 
This  information  must  be  recorded  in  the 
Program  Resource  Tracking  Survey  no  later 
than  10  calendar  days  after  the  month  in 

which  the  document  or  event  is  completed. 


Correspondence  Files 

These  files  include  the  standard 
correspondence  generated  in  the 
administrative  operation  of  MCOTEA. 
This  correspondence  includes  awards, 
personnel  action  requests,  and  some 
fiscal  correspondence. 


MEMORANDUM  FOR  THE  RECORD  3980/XX/MCOTEA 

Date: 

From:  Government  Test  Lead 

Subj:  CLOSURE  OF  PROGRAM  FILE  FOR  [Program  Name  (Aero)] 

As  of  this  date,  [the  program  has  finished  test  and  evaluation].  OR  [further  activity  on  the 
program  has  ceased  due  to  [XXXX]. 


Storage  and  Disposition 

Correspondence  files  are  temporary.  In 
most  cases,  they  will  be  destroyed  after 
2  years  (see  Navy  Records  Management 
Program  Records  Management  Manual 
for  disposition  instructions).  These  files 
are  organized  by  calendar  year  and  by 
the  MCOTEA  SSIC  that  originated 
the  correspondence.  At  the  beginning 
of  each  calendar  year,  new  folders  are 
created  for  each  MCOTEA  SSIC,  and 
all  correspondence  files  signed  after  1 
January  are  placed  in  the  file  for  that  year. 

In  addition  to  storing  records  in  the 
T&E  Reference  Center,  MCOTEA 
tracks  the  completion  of  the  documents 


SIGNED  UPON  PROGRAM  CLOSURE 


[Name] 

[Rank] 

Government  Test  Lead 


1  HAVE  REVIEWED  THE  CONTENTS  OF  THE  T&E  REFERENCE  CENTER  PERTAINING  TO  DOCUMENTATION  AND  MATERIAL 
SUPPORTING  THE  [PROGRAM  NAME]  PROGRAM  AND  CERTIFY  THAT  ALL  REQUIRED  INFORMATION  AVAILABLE  FOR 
STORAGE  HAS  BEEN  PROPERLY  STORED. 

SIGNED  45  DAYS  AFTER  PROGRAM  CLOSURE 


[Name] 

[Rank] 

Government  Test  Lead 


Figure  5-4. 
Sample 

Program  Closure 
Memorandum 
Filed  with  S-1 
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Table  5-4.  Program  Resource  Tracking  Survey  Documents  and  Events 


MCOTEA  Document 

Test  Event 

Endorsement  Letter 

Initial  Operational  Test 

Memorandum  (MFR/MOA/MOU) 

Multi-Service  Operational  Test 

System  Evaluation  Plan 

Follow-on  Operational  Test 

System  Assessment  Plan 

Operational  Assessment 

Test  Concept 

Early  Operational  Assessment 

Test  and  Evaluation  Master  Plan 

DT  w/Marines 

Observation  Plan 

DT  w/o  Marines 

Observation  Report 

Other  MCOTEA-led  Test 

Test  Plan/Event  Design  Plan 

Test  Report 

System  Assessment  Report 

Intermediate  Assessment  Report 

Operational  Test  Agency  Assessment  Report 

Operational  Test  Agency  Milestone  Assessment  Report 

Operational  Test  Agency  Evaluation  Report 

Operational  Test  Agency  Follow-on  Evaluation  Report 

LFT&E  Report 

M&S  Validation/Verification  Plan 

M&S  Accreditation  Plan 

M&S  Validation/Verification  Report 

M&S  Accreditation  Report 

Policy  Letters 

These  are  letters  issued  by  the  Director  that 
establish  policy  and  business  practices  at 
MCOTEA. 

Storage  and  Disposition 

Policy  Letters  must  be  retained  by,  and 
readily  available  in,  the  S-l  until  they  are 
canceled,  overruled,  or  incorporated  into 
other  MCOTEA  guidance,  such  as  this 
manual.  In  addition,  they  have  historical 
significance  to  MCOTEA  and  are 
therefore  maintained  locally  as  permanent 
files.  They  are  organized  as  either  current  or 
historical  (historical  means  the  policy  is  no 
longer  in  effect  or  has  been  incorporated 


into  other  guidance)  and  are  further 
organized  in  chronological  order. 

Review 

The  S-l  ensures  an  annual  review  of  all 
policy  letters  to  determine  which  remain 
current.  If  a  policy  letter  is  canceled,  the 
S-l  draws  a  diagonal  line  in  red  ink  across 
the  first  page  of  the  policy  letter  and  writes 
“Canceled,”  also  in  red  ink.  Canceled  policy 
letters  are  maintained  in  chronological 
order  in  the  Historical  Policies  file. 

Memoranda  of  Agreement  and 
Understanding 

These  documents  constitute  agreements 
between  MCOTEA  and  external 
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organizations,  signed  by  all  adherents. 

ss/c 

Because  only  the  Director  can  commit 
MCOTEA  to  an  agreement  with  an 
external  organization,  MOAs  and  MOUs 
are  assigned  the  Director’s  SSIC. 

Storage  and  Disposition 

MOAs  and  MOUs  are  organized  as 
either  current  or  historical  (historical 
meaning  that  the  MOA/MOU  has  been 
terminated).  Current  MOAs  and  MOUs 
are  organized  alphabetically  by  the  name  of 
the  external  organizations  with  which  the 
agreement  is  made, and  remain  open  files. 

When  the  MOA  or  MOU  is  terminated, 
the  S-l  marks  the  file  “Closed”  on  the 
date  the  MOA  or  MOU  was  deemed 
terminated.  Terminated  MOAs  and  MOUs 
are  of  historical  interest  to  MCOTEA  and 
are  maintained  locally  as  permanent  files 
in  chronological  order  in  the  Historical 
MOU/MOA  file. 

Review 

MOAs  and  MOUs  are  reviewed  annually 
to  ensure  that  they  remain  current.  When 
in  doubt  the  S-l  should  consult  the 
Scientific  Advisor  to  determine  if  an  MOA 
or  MOU  is  current. 

Historical  Files 

Although  they  are  not  official  records,  these 
files  are  composed  of  any  document  or 
other  media  deemed  historically  significant 
to  MCOTEA.  They  include  at  a  minimum 
the  Command  Chronology  submissions 
and  unit  awards,  historical  policies,  and 
MOU/MOAs.  They  may  also  include 
photo  albums,  special  event  pamphlets, 
and  unit  logbooks.  These  files  need  not  be 
assigned  an  SSIC  and  are  not  maintained 
as  official  records. 

Storage  and  Disposition 

The  S-l  maintains  at  least  one  file  drawer 
for  historically  significant  files  and  media. 

All  files  and  media  are  organized  by  date. 
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Annex  A:  Transferring  Records 


Transferring  Records  to 
the  Washington  National 

Records  Center 

OT&E  Program  Files  are  permanent 
records.  SECNAVINST  5210.1  states  that 
these  files  will  be  retired  to  the  Washington 
National  Records  Center  (WNRC)  3  years 
after  program  closure.  In  addition  to  this 
requirement,  MCOTEA  retains  copies  of 
all  program  documentation  and  data  files 
for  20  years  in  accordance  with  this  manual. 
This  annex  describes  how  to  transfer  files  to 
the  WNRC,  located  in  Suitland,  MD. 

Current  information  about  the  records 
center  is  found  on  their  website  at  www. 
archives.gov/  dc-metro/ suitland.  The 
WNRC  provides  records  management 
services  to  headquarters  and  field  offices  of 


federal  agencies  located  in  the  District  of 
Columbia,  Maryland,  Virginia,  and  West 
Virginia.  WNRC  is  the  first  stop  for  Federal 
records  after  they  are  no  longer  actively  used 
by  the  agency  of  origin.  Agency  records 
stay  at  the  WNRC,  where  they  are  tracked 
through  an  automated  database,  until  they 
are  either  destroyed  through  recycling 
or  accepted  by  the  National  Archives 
and  Records  Administration  (NARA)  as 
permanent  records.  All  Federal  agency 
records  management  and  interaction  with 
the  facility  is  governed  by  the  Code  of 
Federal  Regulations  as  it  relates  to  records 
management.  Access  to  most  records  stored 
at  the  facility  is  controlled  by  the  agency  of 
origin;  however,  court  records  are  open  to 
the  public. 


The  following  instructions  for  records  transfer  are 
reprinted  from  www.archives.gov: 


Records  centers  are  authorized  to  store  records 
of  a  Federal  agency  that  are  properly  covered 
by  a  NARA-approved  records  disposition 
schedule  or  the  General  Records  Schedule 
(GRS).  Before  transferring  records  to  a 
records  center,  separate  the  records  into  series. 
A  series  is  defined  as  a  “block  of  records 
having  the  same  disposition  authority  and 
same  disposition  date”(SSIC  codes  and 
program  files).  Each  item  or  subordinate  item 
in  your  record  schedule  represents  a  series. 
Identify  and  separate  your  records  into  blocks 
(series)  by  records  schedule  item  number  and 
cutoff  date.  Transfer  each  series  as  a  separate 
transfer.  Each  transfer  must  consist  of  at  least 
one  box  and  normally  only  one  closing  year 
date  for  a  series  of  temporary  records. 

Filling  out  the  Standard  Form  135 

The  Records  Transmittal  and  Receipt, 
SF-135s,  for  permanent  records  must  be 


accompanied  by  a  detailed  folder  title  list. 
These  lists  may  be  made  on  the  SF-135  itself 
or  on  plain  paper  included  as  an  attachment. 
Agency  offices  may  choose  to  transmit  the 
SF-135  (and  box  listings)  electronically  using 
e-mail.  You  may  obtain  an  electronic  version 
of  the  SF-135  by  visiting  http ://www. archives, 
gov/  frc/  forms/ sf- 1 35  -intro.html 

Approving  the  Standard  Form  135 

The  records  center  staff  will  review  your 
SF-135  for  completeness  and  accuracy.  If 
acceptable,  the  center  will  assign  the  transfer 
number  and  return  one  copy  of  the  SF-135 
within  ten  working  days  authorizing  shipment 
of  the  boxes.  If  you  submitted  a  SF-135 
electronically,  the  “original”  SF-135  can  be 
placed  in  the  first  box  as  the  “shipment” 
copy.  If  the  boxes  or  other  containers  are 
tightly  sealed,  place  this  shipment  copy  in 
an  envelope  taped  to  the  outside  of  the  first 
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container.  All  transfers  must  include  a  copy 
of  the  SF-135  in  box  number  one  of  each 
transfer.  Agencies  should  always  retain  a 
copy  of  the  detailed  box  listing  in  your  office 
so  that  you  may  provide  agency  box  numbers 
when  requesting  reference  service.  Center 
staff  will  return  a  signed  copy  of  the  SF-135 
to  you,  after  the  records  have  been  shelved 
and  issued  a  records  center  location  number, 
as  an  official  receipt.  This  receipted  copy  is 
your  official  record  of  the  transfer  and  should 
be  retained  in  your  files. 

Packing  the  Records 

It  is  wise  to  leave  a  1-2  inch  space  in  each 
box  to  allow  ease  of  reference.  Never  put 
additional  material  on  the  bottom,  side,  or 
top  of  the  records  in  the  box.  Do  not  include 
mixed  media  (e.g.,  computer  diskettes, 
microfilm,  or  videocassettes)  in  the  same 
transfer  with  paper  records  without  prior 
approval  from  the  records  center.  Do  not  over 
pack  the  boxes. 

Numbering  Boxes  for  Shipment 

After  you  receive  the  approved  SF-135  from 
the  records  center,  write  the  transfer  number 
and  the  box  number  in  the  designated  printed 
blocks  on  each  box.  Use  a  black  felt  tip  marker 
and  make  the  numbers  at  least  1.5”  high. 

Do  not  write  on  sealing  tape.  Do  not  place 
tape  over  transfer  or  box  numbers.  For  boxes 
without  the  printed  blocks,  write  the  transfer 
number  in  the  upper  left  corner  and  the 
agency  box  number  in  the  upper  right  corner 
on  one  end  of  each  box.  Begin  with  box 
number  1  and  include  the  total  number  in  the 
transfer,  such  as  1/10, 2/10,  and  so  forth.  Do 
not  use  labels  to  supply  additional  identifying 
information.  No  standard  method  of  affixing 
labels  is  effective  for  long-term  storage.  The 
sides  of  the  boxes  may  be  used  to  write  any 
information  concerning  box  content. 

Shipping  of  Records 

Agencies  are  urged  to  arrange  for  the 
shipment  of  their  records  within  90  days 
after  receipt  of  the  approved  SF-135.  If  the 
transfer  cannot  be  made  within  this  period, 


promptly  advise  center  staff.  Unexplained 
delays  of  more  than  90  days  may  result  in  the 
records  center  canceling  the  transfer  number 
and  returning  your  SF-135.  In  most  instances, 
especially  commercial  transportation  or 
shipment  via  the  U.S.  Postal  Service,  the  boxes 
must  be  sealed  with  tape.  Do  not  tape  over  the 
transfer  number  or  the  agency  box  number. 
For  questions  regarding  shipping  methods 
and  costs,  contact  GSA’s  regional  Traffic  and 
Travel  Service  offices. 

Agencies  may  send  their  records  by  mail, 
FedEx,  United  Parcel  Service  (UPS),  or 
common  carrier  on  pallets.  Some  centers  will 
pick  up  agency  records.  Check  with  your  local 
center  for  scheduling  and  fees.  For  shipments 
of  less  than  20  boxes,  agencies  will  find  it 
most  economical  to  mail  them  to  the  records 
center  or  ship  them  via  UPS.  UPS  shipment 
has  the  advantage  of  automatic  registration 
and  tracing.  For  shipments  over  20  boxes, 
make  all  the  necessary  arrangements  to  ensure 
that  the  boxes  arrive  at  the  records  center  in 
numerical  order  so  that  Box  1 ,  with  a  copy  of 
the  SF-135  included,  is  the  first  box  unloaded. 
If  shipments  of  20  boxes  or  more  must  be 
mailed,  they  may  be  sent  in  a  postal  container 
or  by  bulk  mail.  Agencies  shipping  their  boxes 
on  pallets  using  a  commercial  carrier  should 
complete  a  Transportation  Services  Order 
(TSO).  Shipments  arriving  at  the  center  out 
of  order,  in  oversize  boxes,  improperly  taped, 
or  improperly  marked,  may  require  extensive 
remedial  effort  and  increased  costs.  These 
costs  are  the  responsibility  of  the  shipping 
agency. 

USMC  Procedures 

In  accordance  with  MARADMIN  072/12 

Archiving  records:  Organizations 
electronically  request  HQMC  approval  to 
transfer  records,  and  subsequent  to  receiving 
approval,  process  and  forward  records  directly 
to  NARA. 

Retrieving  Records:  Only  HQMC  can 
withdraw  records  stored  at  NARA. 
Organizations  are  required  to  submit  their 
request  via  the  HQMC  SharePoint  portal. 
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6-  7.  Live  Fire  Test  and  Evaluation 


Public  law  mandates  that  major  weapon 
system  and  munitions  programs,  as  well  as 
product  improvements  to  those  programs 
that  are  likely  to  significantly  affect  the 
vulnerability  or  lethality  of  those  programs, 
undergo  a  realistic  Live  Fire  Test  and 
Evaluation  (LFT&E)  program. 

Simply  put,  LFT&E  is  the  realistic  testing 
of  platforms  or  munitions  against  real 
threats  expected  to  be  encountered  in 
combat.  The  basis  of  the  evaluation  for 
LFT&E  characterizes  the  system  under 
test  against  the  current  and  future  threat 
environment. 

This  section  provides  the  Marine  Corps 
process  for  LFT&E  programs.  It  presents 
the  basis  for  determining  whether  an 
LFT&E  program  is  required  for  a 
given  system,  delineates  the  two  types 
of  LFT&E  programs,  outlines  the  key 
steps  in  developing  an  adequate  LFT&E 
strategy,  and  describes  the  key  building 
blocks  of  LFT&E. 

A  realistic  LFT&E  building  block 
program  represents  the  best  alternative 
to  “actual”  combat  in  assessing  the 
system’s  performance;  however,  with  the 
lack  of  actual  combat  data  a  disciplined 
and  realistic  approach  to  assessing  the 
vulnerability  and  lethality  of  weapons 
systems  must  be  articulated.  A  well- 
planned  and  well-structured  LFT&E 
program  reduces  the  potential  for  surprises 
on  the  battlefield. 

An  early,  active,  well-planned,  well- 
managed,  and  well-executed  LFT&E 
program  is  essential  to  understanding  the 
system.  It  is  also  essential  for  supporting 
decisions  regarding  the  system’s  acquisition 
as  well  as  the  development  of  tactics, 
techniques,  and  procedures  for  its  proper 
operational  employment.  A  properly 
structured  and  integrated  LFT&E 
program,  within  the  overall  context  of 


the  system  T&E  strategy,  will  enable 
design  changes  to  be  incorporated  into  the 
system  at  the  earliest  possible  date,  thereby 
reducing  the  need  for  expensive  retrofit 
programs. 

Objective 

LFT&E  supports  a  timely  and  thorough 
system  vulnerability/lethality  assessment 
during  development  and  subsequent 
production  phases.  It  should  demonstrate 
the  weapon  system’s  or  munition’s  ability 
to  provide  battle-resilient  survivability  or 
lethality.  LFT&E  should  provide  insights 
into 

♦  the  principal  damage  mechanisms  and 
failure  modes  for  the  platform/target 
occurring  as  a  result  of  the  munition/target 
interaction 

♦  techniques  for  reducing  personnel  casualties 
or  enhancing  system  survivability/lethality 

Data  that  emerges  can  be  used  to  support 
cost-effectiveness  trade-offs  to  predict  the 
optimal  “mix”  of  vulnerability/lethality 
enhancement  measures  as  early  as  possible 
in  the  acquisition  cycle. 

The  primary  emphasis  of  LFT &E  is 
testing  under  realistic  combat  conditions 
as  a  source  of  personnel  casualty,  system 
vulnerability,  and  system  lethality 
information  to  ensure  potential  design 
flaws  are  identified  and  corrected  before 
full-rate  production.  The  LFT&E  program 
should  assess  a  system’s  vulnerability/ 
lethality  performance  relative  to  the 
expected  spectrum  of  battlefield  threats; 
it  is  not  constrained  to  addressing  specific 
design  performance  goals  or  threats. 
LFT&E  should  also  assess  the  battle 
damage  assessment  and  repair  (BDAR) 
capabilities  to  enhance  system  survivability. 

Requirement  for  LFT&E 

Public  law  requires  LFT&E  on  “covered” 
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systems  before  proceeding  beyond  low- 
rate  initial  production  (LRIP).  A  “covered” 
system  is  defined  as  a  system  which 
provides  protection  to  users  in  combat  or  is 
considered  a  “major”  system.  A  system  shall 
be  considered  a  “major”  system  if  one  of  the 
following  categories  is  met: 

♦  Total  expenditures  for  research, 
development,  test,  and  evaluation  for  the 
system  are  estimated  to  be  more  than  $365 
million  (in  FYOO  dollars) 

♦  Total  expenditures  for  procuring  the  system 
is  estimated  to  be  more  than  $2,190  million 
(in  FYOO  dollars) 


new  systems  and  changes  (modifications, 
upgrades,  or  follow-on  blocks)  to  existing 
systems.  Additionally,  recent  Defense 
Authorization  Acts  have  included  language 
that  specifically  calls  for  the  LFT&E 
of  equipment  not  normally  subject 
to  LFT&E,  e.g.,  Personal  Protection 
Equipment  (PPE)  such  as  helmets  and 
tactical  vests.  PPE  items  are  now  covered 
under  LFT&E  as  “special  interest” 
programs.  LFT&E  programs  are  subject  to 
DOT&E  oversight. 


♦  The  Secretary  of  Defense  or  the  Secretary 
of  the  Navy  designates  it  as  such  (special 
interest). 


A  “major”  munitions 
program  meets  one 
of  the  above  criteria 
or  has  plans  to 
acquire  more  than 
1,000,000  rounds. 

Specifically, 
the  legislation 
requires  side  by 
side  vulnerability 
LFT&E  if  a  wheeled 
or  tracked  armor 
vehicle  is  to  replace 
an  existing  vehicle; 
requires  LFT&E 
for  all  covered 
systems  and  major 
munition  and 
missile  programs; 
and  requires 
LFT&E  for  product 
improvements 
to  major  systems 
(modification  or 
upgrades).  Figure 
6-1-1  depicts 
the  process  for 
determining  a 
system’s  LFT&E 
requirement  and 
addresses  both 


System  Change  Start 


MCOTEA  Notifies 
OSD  of  LFT&E 
Systems 


Materiel  Developer 

Materiel  Developer 

Identifies  No  LFT&E 

Identifies  LFT&E 

Requirement  to 

Requirement  to 

MCOTEA 

MCOTEA 

Figure  6-1-1. 
LFT&E 

Requirements 

Flowchart 
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Figure  6-1-2. 
Elements  of 
Survivability 


Vulnerability  LFT&E 
deals  with  the  "...testing 
for  vulnerability  of  the 
system  in  combat  by 
firing  munitions  likely  to 
be  encountered  in  combat 
(or  with  a  capability 
similar  to  such  munitions) 
at  the  system  configured 
for  combat,  with  the 
primary  emphasis  on 
testing  vulnerability  with 
respect  to  potential  user 
casualties...and  combat 
performance  of  the 
system"  (Major  systems 
and  munitions  programs 
2008). 
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Vulnerability  LFT&E 

LFT&E  comprises  two  major  components: 
vulnerability  and  lethality.  Vulnerability 
LFT&E  focuses  most  specifically  on  the 
system’s  response  once  a  threat  affects  the 
system,  i.e.,  penetration  and  kill,  which  is 
depicted  by  the  inner  layers  of  figure  6-1-2. 

Penetration 

Penetration  involves  the  actual  defeat  of 
the  platform  protection  system,  normally 
the  armor.  Armor  systems  are  designed 
to  meet  a  protection  specification,  which 
is  normally  delineated  as  the  defeat  of  a 
certain  round  or  munition.  For  example: 
“The  vehicle  will  provide  protection  against 
the  7.62  mm  round  at  zero  degrees  of 
elevation  at  any  azimuth  at  the  muzzle 
velocity.”  However,  LFT&E  is  not 
specification-focused  testing.  LFT&E 
addresses  all  realistic  threats  likely  to  be 


encountered  on  the  battlefield;  as  such, 
the  platform  is  subject  to  “overmatching” 
threats.  While  preliminary  specification- 
based  validation  testing  will  confirm 
baseline  requirements  compliance,  LFT&E 
evaluates  other  rounds  and  determines 
the  conditions  and  distances  from  which 
these  other  rounds  are  able  to  penetrate 
the  platform.  Overmatching  is  the  term 
used  to  describe  testing  against  realistic, 
real-world  threats  that  are  known  to  exceed 
the  baseline  requirements.  In  this  example, 
although  the  overmatched  weapon  would 
be  expected  to  penetrate  the  armor,  the 
test  is  performed  to  determine  the  level  of 
functionality  the  system  retains,  as  well  as 
the  number  and  nature  of  injuries  incurred 
after  penetration.  This  data  can  then  used 
to  adjust  the  system  to  mitigate  the  effects 
of  the  overmatched  threats.  This  helps 
address  one  of  the  goals  of  Vulnerability 
LFT&E — the  characterization  of  the 
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platform’s  armor  system’s  overall  resistance 
to  penetration. 

Kill 

The  “kill”  of  a  system  or  embarked 
crew/personnel  refers  to  the  resultant 
damage  from  a  threat  penetration.  After 
a  round  penetrates  a  system,  several 
damage  mechanisms  affect  platform- 
critical  functionality,  such  as  mobility, 
firepower,  communication,  etc.  There  are 
also  several  distinct  and  concomitant 
damage  mechanisms  that  affect  personnel 
survivability  (Force  Protection). 
Vulnerability  LFT&E  examines  how 
a  platform  mitigates  post-penetration 
damage  mechanisms  such  as  behind  armor 
debris  (BAD),  spall,  ballistic  penetration 
(the  round  itself),  secondary  projectiles, 
toxic  fumes,  shock  and  acceleration, 
and  fire.  Another  goal  of  Vulnerability 
LFT&E  is  to  characterize  a  system’s  loss 
of  functionality  and  embarked  crewmen/ 
personnel  incapacitation  after  the  platform 
has  been  breached  by  a  threat. 

Lethality  LFT&E 

Lethality  LFT&E,  the  less  common  of 
the  two  types  of  LFT&E  programs,  is 
concerned  with  the  system’s  offensive 
capabilities.  Lethality  LFT&E  deals 
with  the  “...testing  for  lethality  by 
firing  the  munition  or  missile 
concerned  at  appropriate 
targets  configured  for 
combat  (Major  systems 
and  munitions  programs 
2008).”  Lethality  is  the 
weapons  system’s  ability 
to  cause  the  loss  of,  or  the 
degradation  in,  the  target 
system’s  ability  to  complete 
its  designated  mission.  In 
requirements  documents, 
lethality  is  normally  delineated 
in  the  form  of  a  “target  set”  or 
“target  list.” This  target  set  outlines 
the  required  targets  and  the  desired 
effect  on  each  target.  For  example:  The 


platform  “will  suppress  infantry  in  the 
open  at  1,000  meters”  or  “will  destroy 
Light  Armored  Vehicles  at  800  meters.” 
The  major  components  of  lethality,  shown 
in  figure  6-1-3,  are  accuracy  and  terminal 
effects.  Typically,  the  effectiveness  of 
accuracy,  (seeing,  acquiring,  and  hitting  the 
target)  is  resolved  during  OT&E  as  part 
of  the  system’s  Operational  Effectiveness. 
Terminal  effects  (penetrating  and  killing 
the  target),  the  inner  two  circles  in  figure 
6-1-3,  are  examined  during  Lethality 
LFT&E.  Ultimately,  an  end  to  end  mission 
profile  using  real  rounds  against  real  threat 
targets  is  normally  conducted  as  part  of 
I  OT&E  lethality  testing.  Lethality  is 
referred  to  as  “Vulnerability  LFT&E 
in  reverse.”  As  such  the  parameters  of 
“penetrate”  and  “kill”  are  presented  in  both. 

LFT&E  Management 

While  the  details  of  each  element  of  an 
overall  LFT&E  program  must  be  decided 
on  a  case  by  case  basis,  this  chapter 
presents  the  general  approaches  and  lessons 
learned  from  previous  successful  Marine 
Corps  LFT &E  programs  and  should 
prove  beneficial  to  those  involved  in  future 
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LFT&E  programs.  Figure  6-1-4  depicts 
the  basic  elements  of  the  overall  LFT&E 
process  from  the  initial  strategy  definition 
to  the  writing  of  the  final  LFT&E  report. 
Before  documenting  issues  to  support 
LFT&E  strategy  development,  the 
LFT&E  analyst  must  obtain  the  COIs  for 
the  system  from  MCOTEA’s  test  team. 
These  COIs  form  the  basis  for  the  critical 


Figure  6-1-4.  LFT&E 
Process  Flow  Chart 
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LFT&E  issues.  The  “Strategy  Review 
Conference”  depicted  in  figure  6-1-4 
constitutes  stakeholder  concurrence  with 
the  overall  LFT &E  strategy. 

Although  current  legislation  only  requires 
LFT&E  in  certain  cases,  it  provides  a 
means  of  ensuring  that  Marines  using 


the  system  in  combat  are  protected  to 
the  highest  degree  possible.  The  scope 
of  LFT&E  needs  to  be  addressed  in 
a  comprehensive  LFT&E  strategy, 
incorporated  into  the  appropriate 
documentation,  and  provided  to  Marine 
Corps  leadership  for  guidance  and 
approval.  According  to  the  Secretary  of 
the  Navy,  the  Naval  materiel  developer  is 
responsible  for  executing  LFT &E 
(Secretary  of  the  Navy  2008).  The 
materiel  developer  is  responsible  for 
completing  the  system’s  LFT&E 
and  ensuring  that  the  LFT&E 
Report  is  completed  and  submitted 
prior  to  a  Full-Rate  Production 
decision.  SECNAVINST  5000.2 
delineates  that  the  materiel  developer 
must  submit  the  LFT &E  Report 
to  DOT&E  via  MCOTEA.  The 
materiel  developer  has  the  following 
options  available  when  addressing 
the  inherent  requirement  to  execute 
LFT&E: 

♦  Task  MCOTEA  to  execute  and 
report  on  LFT&E;  historically  this 
is  the  preferred  technique  to  conduct 
LFT&E  within  the  Marine  Corps.  In 
this  arrangement,  MCOTEA  chairs  the 
LFT&E  IPT. 

♦  Execute  LFT&E  with  MCOTEA 
oversight;  historically  this  option  has 
been  taken  with  minor  “special  interest” 
LFT&E  programs.  In  this  arrangement 
MCOTEA  co-chairs  the  LFT&E  IPT 

♦  Task  an  outside  technical  agent/ 
agency  to  conduct  LFT&E  on  behalf  of 
the  materiel  developer;  this  is  the  least 
preferred  method  and  involves  both 
MCOTEA  oversight  and  the  inclusion  of 
an  outside  agency,  which  may  or  may  not 
have  the  requisite  experience  in  LFT&E. 

MCOTEA  typically  acts  as  co-chair  of  the 
LFT&E  IPT  in  this  arrangement. 

The  system’s  proposed  acquisition  strategy 
and  overall  evaluation  strategy  should 
include  live  fire  testing  requirements  with 
supplementary  and  complementary  data  to 
be  drawn  from  DT  and  OT.  The  system’s 
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mature  LFT&E  strategy  and  resource 
requirements  should  be  included  in  the 
systems  TEMP.  The  programs  LFT&E  IPT 
develops  and  produces  the  LFT&E  strategy. 
The  LFT&E  IPT  produces  and  reviews 
all  LFT&E  documents.  Typically  there  are 
several  core  members  of  the  LFT&E  IPT 
including  representatives  from 

♦  PM/Materiel  Developer 

♦  MCOTEA 

♦  Marine  Corps  Intelligence  Activity 

♦  DC,  CD&I 

♦  Technical  test  agency  (normally  Aberdeen 
Test  Center  for  most  Marine  Corps  ground 
systems,  and  the  M&S  agent  (normally 
Army  Research  Lab  (ARL)) 

♦  DOT&E 

MCOTEA  Vulnerability  Process 

Live  fire  consists  of  a  range  of  testing  and 
evaluation  that  begins  with  preliminary 
component,  subsystem,  and/or  system-level 
tests  and  culminates  in  Full-Up  System 
Level  (FUSL)  tests  of  system  vulnerability 
and  lethality.  FUSL  live  fire  testing  satisfies 
the  statutory  requirement  for  “realistic 
survivability  testing”  or  “realistic  lethality 
testing. ’’The  LFT&E  program  includes 
all  vulnerability/lethality  T&E  phases  and 
associated  modeling  and  analysis  efforts 
that  support  the  live  fire  evaluation. 

Vulnerability  LFT&E  focuses  on 
protection  against  lethal  mechanisms 
and  minimizing  damage  to  the  crew  and 
hardware  given  an  impact  or  breach  by  a 
lethal  mechanism.  In  addition,  vulnerability 
LFT&E  addresses  recoverability  from 
combat  damage.  Critical  issues  for 
Vulnerability  LFT&E  address  the 
following  key  areas: 

♦  Crew/Occupant  vulnerabilities  (Force 
Protection) 

♦  System  and  hardware  vulnerabilities 
(Vehicle  Survivability) 

♦  BDAR  capabilities 


Table  6-1-1  contains  MCOTEA’s  LFT&E 
process  and  milestones  within  a  generic 
LFT&E  vulnerability  program.  The 
LFT&E  IPT  chair  executes  the  checklist 
shown  in  table  6-1-1;  however,  because 
each  program  is  unique,  certain  elements 
will  not  apply  to  every  LFT&E  strategy. 
Regardless  of  the  scope  for  the  LFT&E 
program,  this  process  serves  as  a  guide  to 
effectively  incorporate  live  fire  testing  into 
the  system’s  overall  T&E  strategy. 

Table  6-1-1.  MCOTEA  Live  Fire  Vulnerability  Process 


Test  Concept 
Development 

□  Review  program  Documentation 

□  Review  requirements/capabilities  Documents 

□  Review  System  Survivability  Specifications 

□  Form  LFT&E  IPT 

o  Designate  Chair  (MCOTEA  either  Chair  or  Co-Chair) 
o  ID  Core  members 
o  Develop/Approve  IPT  Charter 

□  Obtain  updated  COIs  from  MCOTEA  OA 

□  Determine  Level  of  M&S  Needed 

Strategy  Review 

□  Present  Draft  LFT&E  concept 

Conference 

1 .  Armor  Validation  Scope 

2.  Armor  Characterization  Scope 

3.  Armor  Exploitation 

4.  Ballistic  Hull  &  Turret  (BH&T)  Scope 

5.  Component  Candidates  for  Component  Ballistic  Testing 

6.  Determine  screening  criteria 

7.  System  Level  test  scope 

8.  Controlled  Damage  Testing  (CDT)  scope 

9.  FUSL  scope 

1 0.  M&S  scope 

□  Assign  agencies  to  conduct  LFT&E  events 

□  Coordinate  need  for  Marine  operating  forces  with  Force  Synchronization  Conference 
(normally  BDAR  participants) 

□  Present  Draft  LF  Critical  issues 

□  Present  Draft  LF  Strategy 

□  Obtain  BDAR  concept  plan  from  PM/Material  Developer 

□  Coordinate  with  MCOTEA  OA  to  ensure  all  LFT&E  OSun  data  requirements  will  be  met 

□  Present  Draft  Live  Fire  System  Evaluation  Plan 

□  Update/Present  funding/resource  profile 

TEMP 

□  Approve  LF  Strategy 

Development 

□  Approve  LF  Critical  issues 

□  Submit  M&S  W&A  plan 

□  Submit  M&S  Requirements 

□  Synchronize  PM/Material  Developer  Armor  Specification  compliance  with  LF  Strategy 
Coordinate  TEMP  inputs  with  the  MCOTEA  System  Evaluator  pertaining  to  components  and 
test  assets  required  for  LFT&E 

□  Submit  Live  Fire  System  Evaluation  Plan 

LFT&E  Test  Plan 

□  Develop,  approve,  and  distribute  the  following  plans  via  the  LFT&E  IPT 

Development 

1 .  Armor  Characterization 

2.  Component  Ballistic  Testing  (CBT) 

3.  Armor  Exploitation 

4.  BH&T 

5.  BDAR 

6.  System  Level  Test 

7.  CDT 

8.  FUSL 

□  Develop  and  submit  M&S  Accreditation  Plan  to  Dir,  MCOTEA  or  designated  representative 

□  Track  BDAR  development  &  insure  BDAR  elements  are  addressed  in  all  applicable  LFT&E  test 
events. 

Execution  of 

□  Observe  LFT&E  events 

LFT&E  Building 

□  Track  and  review  M&S  W  report 

Block  events. 

□  Report  results  to  MCOTEA  OA 

FUSLTRR 

□  Insure  test  Asset  availability 

□  Arrange  for  system  testing  for  Marine  participants  who  will  conduct  Post  shot  functionality 
testing  and  BDAR 

□  Receive  preliminary  M&S  Accreditation 

FUSL  Test 

□  Observe  and  Monitor  FUSL  conduct 

Execution 

□  Obtain  Pre-shot  predictions  (normally  from  M&S  stakeholder) 

□  Compare  Pre-shot  predictions  with  actual  outcomes 

□  Provide  updates/SITREPs  to  Dir,  MCOTEA 

□  Report  results  to  MCOTEA  OA 

Conduct  DAT 

□  Convene  Damage  Assessment  Team 

activities 

□  Receive  final  M&S  Accreditation 

Produce  LFT&E 

□  Collect  and  Review  all  applicable  DT  and  LFT&E  Reports.  Report  results  to  MCOTEA  OA 

Report 

□  Receive  and  Review  MUVES  S-2  Model  effort  output 

□  Submit  final  draft  of  USMC  LFT&E  Report  for  MCOTEA  Content  Review  Board 

□  Resolve  any  Ballistic  requirements  for  OT&E  OER 

□  Publish  and  Route  USMC  LFT&E  Report  (Report  needs  to  be  delivered  to  DOT&E  45  days  prior 
to  a  Full  Rate  Production  decision) 

Building  Block  Approach 

The  building  block  approach  helps  build 
upon  the  system’s  sequential  LFT&E. 

This  information,  especially  early  in  the 
life  of  a  program,  helps  shape  and  improve 
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system  design.  The  main  building  blocks 
in  a  Vulnerability  LFT&E  program  are 
listed  chronologically  but  can  be  repeated  if 
necessary  and  are  defined  in  the  following 
sections. 

Armor  Validation 

Armor  Validation  is  normally  a  DT, 
materiel  developer-conducted  set/series 
of  tests  executed  at  the  armor  coupon 
level  to  determine  if  the  armor  solution 
meets  its  technical  specifications.  Coupon 
testing  involves  testing  an  isolated  piece  of 
armor  on  its  own  when  not  incorporated 
in  to  the  overall  system.  While  a  DT 
event,  the  LFT&E  IPT  will  observe 
this  test  and  receive  copies  of  the  test 
reports.  The  LFT&E  IPT  may  require 
additional  coupon  tests  that  examine  the 
environmental,  multihit,  and  durability 
characteristics  of  the  armor.  Depending 
on  the  platform’s  intended  operational 
environment,  additional  coupon  tests  may  be 
required  to  ascertain  the  limit  of  resistance 
to  penetration  for  designated  overmatching 
threats.  The  results  of  coupon  testing  will 
be  used  to  build  the  baseline  resistance  to 
the  penetration  module  in  the  M&S  suite 
and  to  ensure  the  vendor’s  specification 
compliance.  Typically,  the  Marine  Corps 
uses  the  Army’s  Modular  Unix-based 
Vulnerability  Estimation  Suite  (MUVES) 
S-2  M&S  tool  for  LFT&E. 

Armor  Characterization 

Armor  Characterization  identifies  the 
Armor’s  BAD  characteristics,  which  is 
often  referred  to  as  the  “spall  cone  angle.” 
Typically,  several  overmatching  threats 
are  examined  and  the  BAD  data  is  then 
transferred  to  the  MUVES  S-2’s  BAD 
module.  Armor  Characterization  also 
defines  the  armor’s  dynamic  deflection 
properties.  Dynamic  deflection  testing 
helps  the  materiel  developer  identify 
the  “safe”  distance  behind  the  armor. 

This  influences  the  placement  of  critical 
components  and  seats  for  occupants. 


MCOTEA’s  active  involvement  in 
LFT&E  typically  begins  with  this  step 
and  continues  through  to  the  end  of  the 
system’s  LFT&E. 

Armor  Exploitation 

Armor  Exploitation  characterizes  an 
armor  system’s  resistance  to  penetration. 
Instead  of  testing  coupons,  the  integrated 
armor  solution  (Pre-MS  B  prototype)  is 
examined.  The  areas  of  interest  are  usually 
armor  seams,  armor  interface  points, 
through  bolts,  and  locking  mechanisms 
embedded  on  or  in  the  armor. 

Ballistic  Hull  and  Turret 

Ballistic  Hull  and  Turret  (BH&T)  testing 
is  typically  performed  on  a  Technology 
Development  Phase  prototype  to 
verify  system-wide  ballistic  protection 
requirements  (usually  underbody/under 
wheelAmder  track  blast  requirement). 

This  is  typically  an  LFT&E  event,  but  the 
materiel  developer  heavily  influences  the 
test  event  design.  Additionally,  BH&T 
may  be  used  to  conduct  preliminary  end 
to  end  Fire  Extinguishing  System  testing 
across  the  entire  platform.  From  an  asset 
allocation  standpoint,  the  BH&T  asset  is 
often  used  for  both  BH&T  and  Armor 
Exploitation.  If  significant  vulnerabilities 
are  discovered  during  these  two  test  phases, 
and  design  improvements  are  made  to 
mitigate  these  vulnerabilities,  BH&T  is 
then  typically  repeated  on  the  follow-on 
design  (normally  a  post-MS  B  platform). 

Component  Ballistic  Testing 

Component  Ballistic  Testing  (CBT) 
examines  the  critical  component’s  ballistic 
properties  within  a  platform.  The  data  from 
this  testing  provides  information  on  the 
specific  component’s  vulnerability  and  also 
provides  the  Probability  of  Component 
Dysfunction  (PCD).  The  PCD  for  specific 
components  is  then  loaded  into  MUVES 
S-2.  During  the  CBT  Phase  a  Criticality 
Analysis  and  Damage  Assessment  List 
are  produced  by  ARL  and  DC,  CD&I 
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respectively.  Hie  Critical  Analysis  addresses 
the  system  component’s  engineering  and 
functional  hierarchy,  while  the  Damage 
Assessment  List  addresses  a  specific 
component’s  critical  functionality  “value” 
(communications,  mobility,  firepower)  of 
the  overall  platform.  Both  the  Critical 
Analysis  and  the  Damage  Assessment  List 
are  inputs  to  MUVES  S-2. 

System-Level  Testing 

System-Level  (SL)  testing  examines 
system-wide  response  to  threat  interactions 
while  accounting  for  threat  tactics. 
Typically,  vulnerabilities  that  were 
uncovered  early  in  the  LFT&E  process 
are  revisited  to  determine  if  design 
improvements  have  mitigated  known 
vulnerabilities.  SL  testing  also  verifies  and 
validates  BDAR  procedures.  This  testing  is 
normally  conducted  on  a  late  Engineering 
and  Manufacturing  Development  Phase 
prototype.  SL  testing  also  allows  the 
PM  to  verify  any  system-level  ballistic 
requirements/specifications  prior  to  MS-C. 
The  evaluation  from  SL  concomitantly 
affords  the  materiel  developer  with  timely 
input  to  help  focus  the  Critical  Design 
Review  before  committing  to  an  LRIP 
design. 

Controlled  Damage  Testing 

Controlled  Damage  Test  (CDT)  is  a  pre¬ 
cursor  FUSL  event  that  occurs  on  the  asset 
before  the  full-up  shots  begin.  CDT,  in  a 
non-destructive  format,  looks  to  verify  the 
Critical  Analysis  and  update  any  changes 
to  the  Critical  Analysis  in  MUVES  S-2 
prior  to  FUSL. 

Full-Up  System-Level  Testing 

Full-up  System-Level  Testing  (FUSL) 
testing  involves  a  complete,  production- 
representative  platform  with  all  ancillary, 
support  equipment,  fuel,  and  ammunition 
onboard.  MUVES  S-2  is  used  to  conduct 
pre-shot  predictions  to  estimate  embarked 
personnel  incapacitation  and  damage  to  the 
vehicle.  The  FUSL  pre-shot  predictions  are 


compared  to  the  actual  damage  incurred 
to  improve  the  fidelity  of  the  model. 

The  Damage  Assessment  Team  assesses 
actual  damage  and  personnel  injury  and 
incapacitation  data  from  a  FUSL  event  and 
subsequently  distributes  the  information  to 
the  LFT&E  IPT. 

Marine  Corps  Lethality  Process 

Lethality  LFT&E  addresses  both  the 
ability  to  perforate  or  breach  the  target  and 
to  inflict  significant  damage  to  the  target 
and/or  its  crew  and  occupants.  Generally, 
the  following  lethal  abilities  are  critical: 

♦  accurately  engage  a  threat  system  (often 
evaluated  using  DT  and  OT  data) 

♦  perforate  or  breach  the  threat  system’s 
protection. 

♦  significantly  degrade  the  threat  system’s 
combat/mission  functions 

♦  injure/incapacitate  the  crew/occupants 

Building  Block  Approach 

The  main  building  blocks  in  a  Lethality 
LFT&E  Program  are  listed  chronologically 
and  defined  in  the  following  sections. 

Qualification  Testing 

Qualification  testing  is  typically  a  DT, 
materiel  developer-conducted  set/series  of 
tests  executed  to  qualify  the  munition  for 
service,  to  safety  certify  the  munition,  and 
to  gain  initial  data  on  its  capabilities. 

Munition  Terminal  Effects 
Characterization 

Munition  Terminal  Effects 
Characterization  is  typically  a  DT,  materiel 
developer-conducted  set/series  of  tests 
executed  from  a  fixed  firing  point  against 
representative  armor  coupons  and  surrogate 
targets  to  determine  if  the  munition  meets 
its  technical  specifications.  While  a  DT 
event,  the  LFT&E  IPT  will  observe 
this  testing  and  receive  copies  of  the  test 
reports.  The  LFT&E  IPT  may  require 
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additional  tests  that  examine  capabilities 
of  the  munition  against  realistic  target  sets 
expected  to  be  encountered  in  combat.  The 
results  of  this  testing  are  used  for  M&S 
purposes  to  build  the  baseline  terminal 
effects/penetration  module  in  the  M&S 
suite.  Typically,  the  Marine  Corps  utilizes 
the  Army’s  MUVES  S-2  model  for 
Lethality  LFT&E.  Ultimately,  the  purpose 
of  this  testing  will  be  to  see  if  an  accurately 
delivered  munition  delivers  the  desired 
terminal  effect. 

System-Level 
Testing 

System  Level  testing 
is  normally  an 
LFT&E  event  that 
looks  to  characterize 
the  lethality  of  a 
munition  when  it  is 
delivered  by  a  host 
platform  against 
realistic  targets. 

Engagement 
TTPs  are  typically 
developed  during  this 
test  series/phase. 

End- to- End  FUSL 
Testing 

End-to-End  FUSL 
Testing  involves  the 
real  munition,  with 
real  Marine  operators,  with  its  intended 
delivery  system,  engaging  realistic  targets  at 
tactically  relevant  distances  to  characterize 
its  operational  lethality.  This  testing  is 
often  executed  within  the  context  of  an 
Operational  Mission  Profile. 

Table  6-1-2  illustrates  the  MCOTEA 
process  and  milestone  events  within  a 
generic  LFT&E  lethality  program.  Since 
each  program  is  unique,  certain  elements 
will  not  apply  to  every  strategy.  Regardless 
of  the  scope  for  the  LFT&E  program, 
this  process  serves  as  a  guide  to  effective 
incorporation  of  live  fire  testing  into  the 


overall  test  and  evaluation  strategy. 

LFT&E  Key  Elements 

Each  live  fire  program  contains  several 
critical  elements,  defined  in  further  detail 
in  this  section. 

Scope  of  LFTEE 

The  following  questions  should  be 
considered  in  order  to  prepare  a  properly 
scoped  LFT&E  strategy: 

♦  What  are  the  technical  and  operational 
characteristics  of  the 
concepts,  technology, 
and  requirements  for  the 
system? 

♦  How  do  they 
differ  from  the  system 
being  replaced  (where 
appropriate)? 

♦  Which  threats  are 
to  be  considered  in  the 
LFT&E? 

The  threats  considered 
in  LFT&E  should 
be  based  on  a  review 
of  the  System  Threat 
Assessment  (STA); 
the  densities  of 
various  classes  of 
threat  weapons  and 
countermeasures  in 
organizations  likely 
to  be  encountered; 
and  the  frequency  that  various  threats 
kill  or  are  killed  by  the  system  from  force 
effectiveness  analysis  supporting  program 
decisions  or  planning  studies. 

LFTElE  Strategy 

The  LFT&E  strategy  is  the  most 
important  element  of  the  LFT&E 
process.  An  LFT&E  concept  should  be 
prepared  and  approved  as  early  as  possible 
in  the  acquisition  cycle,  with  the  goal  of 
producing  a  viable  LFT&E  Strategy  by 
MS  B.The  LFT&E  System  Evaluator 
has  the  lead  in  preparing  and  obtaining 


Vulnerability  LFT&E 

Breaks  threats  into  major  and  minor 

•  A  major  threat  kills  or  reduces  the  effectiveness 
of  a  large  percentage  of  the  systems  in  the  force 
effectiveness  evaluation  or  has  a  high  density 
in  the  force 

•  all  other  threats  are  considered  minor 
PM  provides  funding 

Lethality  LFT&E 

The  MCIA  representative  to  the  LFT&E  IPT  plays  a 
key  role  in  identifying  potential  realistic  targets  to 
the  LFT&E  IPT 

Based  on  the  target  driving  the  design  (usually  the 
most  difficult  target  to  kill  given  a  hit) 

Identifies  threat  target  reguirements  and 
availability 

PM  provides  funding  and  acguires  targets 
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approval  for  the  strategy  in  coordination 
with  the  LFT&E  IPT.  The  ACMC 
approves  the  strategy  for  the  Marine 
Corps  before  it  is  sent  (via  the  TEMP)  to 
DOT&E  for  approval.  If  consensus  cannot 
be  reached  on  the  LFT&E  scope,  or  if 
program  constraints  limit  compliance  with 
required  reporting  dates,  the  ACMC  will 
be  approached  to  help  resolve  the  issue. 

The  LFT&E  strategy  is  the  foundation  of 
the  LFT&E  section  of  the  TEMP  and  all 
subsequent  planning  documents  (Live  Fire 
System  Evaluation  Plan  (LFSEP),  Event 
Design  Plan  (EDP),  Pre-Shot  Prediction 
Report,  and  the  Test  Plan).  The  strategy 
should  be  detailed  enough  to  adequately 
project  resource  requirements;  schedules 
for  major  T&E  efforts;  and  to  trigger  long 
lead  time  planning,  procurement  of  threats/ 
surrogates,  and  modeling. 

How  LFT&E  results  will  be  evaluated  is 
formulated  during  the  system  vulnerability/ 
lethality  examination  and  during  the 
definition  of  critical  LFT&E  issues.  After 
strategy  development,  the  evaluation 
process  is  finalized  the  and  the  details  are 
articulated  in  the  LFSEP  and  LFT&E 
EDP.  The  evaluation  must  crosswalk 
all  vulnerability/lethality  testing  and 
complementary  modeling  and  assessment 
with  LFT&E  issues.  The  following  aspects 
of  the  evaluation  process  must  be  examined 
when  developing  the  LFT&E  strategy: 

♦  Possibly  using  M&S  to  address  evaluation 
issues  pertaining  to  system  vulnerability 
or  lethality,  crew  casualties,  and  logistics 
supportability. 

♦  Planning  building  block-level  vulnerability 
tests  to  assess  the  protective  system  of  the 
item  under  test’s  ability  (for  example,  armor 
and  optics)  to  withstand  impacts  by  threat 
missiles  and  projectiles,  and  to  examine  the 
ability  of  critical  components  (for  example, 
ammunition  compartments)  to  withstand 
damage  from  a  threat  warhead  or  projectile 
that  breaches  the  protective  system.  Early 
system  development  LFT&E  will  focus  on 
the  component/subsystem  level  to  address 


Table  6-1-2.  MCOTEA's  Live  Fire  Lethality  Process 


Test  Concept 
Development 

□  Review  program  Documentation 

□  Review  requirements/capabilities  Documents 

□  Review  System  Lethality  Specifications  and  Target  list 

□  Form  LFT&E  IPT 

o  Designate  Chair  (MCOTEA  either  Chair  or  Co-Chair) 
o  ID  Core  members 
o  Develop/Approve  IPT  Charter 

□  Obtain  updated  COIs  from  MCOTEA  OA 

□  Determine  Level  of  M&S  Needed 

Strategy  Review 
Conference 

□  Present  Draft  LFT&E  concept 

□  Determine  screening  criteria 

□  Assign  agencies  to  conduct  LFT&E  events 

□  Present  Draft  LF  Critical  issues 

□  Present  Draft  LF  Strategy 

□  Present  Draft  Live  Fire  System  Evaluation  Plan 

□  Coordinate  need  for  Marine  operating  forces  with  Force 
Synchronization  Conference 

□  Update/Present  funding/resource  profile 

TEMP 

Development 

□  Approve  LF  Strategy 

□  Approve  LF  Critical  issues 

□  Submit  M&S  W&A  plan 

□  Synchronize  PM/Material  Developer  Lethality  Specification 
compliance  with  LF  Strategy 

□  Coordinate  TEMP  inputs  with  the  MCOTEA  OA  pertaining  to 
components  and  test  assets  required  for  LFT&E 

□  Submit  Live  Fire  System  Evaluation  Plan 

LFT&E  Test  Plan 
Development 

Develop,  approve,  and  distribute  the  following  plans  via  the  LFT&E  IPT 
o  Qualification  Testing 
o  Munition  Terminal  Effects  Characterization 
o  System  Level  Test  Plan 

o  End-to-End  FUSL  Test  Plan  (normally  executed  during  IOT&E) 

□  Develop  and  submit  M&S  Accreditation  Plan  to  Dir,  MCOTEA  or 
designated  representative 

Execution  of  LFT&E 
Building  Block 
events. 

□  Observe  LFT&E  events 

□  Track  and  review  M&S  W  report 

□  Report  results  to  MCOTEA  OA 

End-to-End  FUSL 
TRR 

□  Insure  test  Asset  availability 

□  Arrange  for  system  testing  for  Marine  participants  who  will  conduct 
End-to-End  FUSL  Testing 

□  Receive  preliminary  M&S  Accreditation 

FUSL  Test 
Execution 

□  Observe  and  Monitor  FUSL  conduct 

□  Provide  Pre-shot  predictions 

□  Compare  Pre-shot  predictions  with  actual  outcomes 

□  Report  results  to  MCOTEA  OA 

□  Provide  updates/SITREPs  to  Dir,  MCOTEA 

Conduct  Target 
DAT  activities 

□  Convene  Damage  Assessment  Team  (DAT)  for  Target  analysis 

Receive  final  M&S  Accreditation 

Produce  LFT&E 
Report 

□  Collect  and  Review  all  applicable  DT  and  LFT&E  Reports 

□  Report  results  to  MCOTEA  OA 

□  Submit  final  draft  of  USMC  LFT&E  Report  for  MCOTEA  Content 

Review  Board 

□  Resolve  any  Ballistic  requirements  for  OT&E  IER 

□  Publish  and  Route  USMC  LFT&E  Report  (Report  needs  to  be 
delivered  to  DOT&E  45  days  prior  to  a  Full  Rate  Production  decision) 

vulnerability  issues  and  upgrade  and  develop 
the  system  vulnerability  model.  The  FUSL 
vulnerability  LFT&E  conducted  against 
a  full-up  (combat-loaded)  production 
or  production-representative  system  is 
generally  the  last  in  the  series  of  live  fire 
tests  conducted. 

♦  Lethality  LFT&E  must  be  planned  to 
assess  the  system’s  ability  to  damage  target- 
critical  components  and  injure/incapacitate 
the  crew.  During  the  early  weapons  system 
development,  the  testing  will  usually  focus 
on  the  warhead’s  or  penetrator’s  ability  to 
breach  the  threat  target’s  protective  system. 
During  pre-qualification  and  qualification 
testing,  impact  conditions  will  be  firmly 
established  for  the  missile  or  projectile  and 
the  ability  of  the  warhead  or  penetrator  to 
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breach  the  target’s  protective  system  will  be 
refined.  The  End-to-End  FUSL  lethality 
life  five  testing  is  the  last  live  fire  testing 
phase  and  is  conducted  against  a  full-up 
(combat-loaded)  threat  target.  However, 
the  extent  of  target  functionality  and 
application  of  combat  load  may  be  impacted 
by  availability  of  assets  and  specific  T&E 
requirements.  If  it  is  not  possible  to  obtain 
a  realistic  threat  target,  FUSL  lethality 
LFT&E  must  use  the  best  available 
surrogate  threat  targets.  The  scarcity  of 
lethality  LFT&E  targets  and  their  cost 
may  dictate  that  these  targets  not  be  fully 
combat-loaded  with  live  munitions  to 
preclude  catastrophic  loss. 

♦  Vulnerability  models  are  also  used  to 
estimate  the  spare  parts  and  time  required 
to  repair  combat  damaged  components. 
FUSL  vulnerability  LFT&E  provides 
valuable  inputs  for  refining  these  estimates. 
In  addition,  rapidly  returning  damaged 
systems  to  battle  requires  accurately 
accessing  the  damage  and  applying 
field-expedient  repairs.  Again,  FUSL 
vulnerability  LFT&E  provides  the  materiel 
developer  and  TECOM  with  valuable 
training  opportunities  to  refine  and  develop 
field-expedient  repair  methods  and  to 
identify  tools  and  materials  required  to 
execute  these  repairs. 

Live  Fire  System  Evaluation  Plan 

Specifically,  the  LFSEP  provides  the 
crosswalk  between  the  evaluation  issues 
and  the  data  requirements.  Additionally, 
the  data  sampling  plan  and  analysis 
techniques  are  specified  to  ensure  the  logic 
of  the  evaluation  is  understandable.  The 
LFSEP  will  identify  MOPs  and  MOEs 
associated  with  the  issues  developed  in  the 
strategy.  The  LFSEP  includes  a  section 
describing  the  types  of  threats  or  targets 
that  the  system  is  expected  to  encounter 
during  the  operational  life  of  the  system 
and  the  key  characteristics  of  the  threats/ 
targets  that  affect  system  vulnerability/ 
lethality.  Any  T&E  limitations  or  shortfalls 
and  their  impact  will  be  discussed. 

The  Event  Design  Plan  (EDP)  contains 


guidance  on  the  conditions  and  data 
requirements  for  use  in  the  development 
of  the  Test  Plans.  The  EDP  is  concerned 
with  the  higher-level  issues  of  interest 
in  constructing  the  Test  Plan  such  as 
vulnerable/physical  areas  to  examine, 
impact  angles,  and  whether  or  not  to 
examine  seams.  The  EDP  also  describes 
statistical  analyses,  criteria,  models,  system 
comparisons,  and  how  they  support  the 
evaluation.  The  EDPs  provide  the  tester 
or  analyst  with  the  details  on  what  data  is 
required  from  a  particular  test  or  evaluation 
event.  The  EDP  will  detail  the  decision 
process  for  foreseeable  changes  in  the  test 
design.  If  an  unexpected  change  in  the  test 
design  is  required,  the  change  to  the  EDP  is 
submitted  to  the  Director,  MCOTEA  for 
approval  90  days  prior  to  test  initiation  and 
is  subsequently  forwarded  to  DOT&E. 

Threats 

An  integral  part  of  the  LFT&E  strategy 
development  is  identifying  the  threat 
target  (lethality  LFT&E)  and  munition 
(vulnerability  LFT&E)  requirements. 
These  requirements  need  to  be  identified 
early  in  the  acquisition  cycle  to  allow  for 
possible  long  lead  times  for  procurement. 

It  is  very  likely  that  some  of  the  required 
threat  munitions  will  not  be  available  for 
LFT&E.  It  is  also  likely  that  intelligence 
data  on  some  munitions  may  be  limited. 
Therefore,  LFT&E  maybe  conducted 
using  threat  munitions  based  on 
postulated  technology  options  derived 
from  intelligence  assessments.  This  will 
require  surrogates  in  lieu  of  “real”  threats. 
The  rationale  for  threat  surrogate  selection 
must  be  detailed  in  the  LFT&E  strategy. 

The  rationale  for  selecting  surrogate  threat 
projectiles  for  vulnerability  LFT&E 
is  to  match  physical  performance 
characteristics  of  the  projected  threat.  For 
kinetic  energy  projectiles,  penetration 
into  rolled  homogeneous  armor  (RHA); 
muzzle  velocity  and  impact  velocity;  and 
penetrator  material,  length,  and  diameter 
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are  typical  key  parameters.  For  shaped 
charge  warheads,  typical  parameters  are 
penetration  into  RHA,  impact  velocity, 
and  warhead  diameter  and  explosive  type. 
Availability  and  cost  of  surrogate  projectiles 
may  also  drive  the  selection.  Typically,  U.S. 
projectiles  and  warheads  will  be  selected 
as  surrogates.  The  projectiles  and  warheads 
selected  as  threat  surrogates  must  be 
submitted,  along  with  supporting  rationale, 
to  the  Director,  MCOTEA  for  approval. 

Shotlines 

The  attack  conditions  and  the  munition/ 
target  impact  location  (that  is,  shotline) 
must  be  identified  for  each  shot  to  provide 
the  appropriate  information  required 
the  address  critical  LFT&E  issues.  The 
shotline  selection  methodology  that  will  be 
used  is  described  in  the  LFT&E  Strategy, 
whereas  the  specific  shotlines  selected 
and  the  rationale  for  their  selection  must 
be  included  in  the  EDP  for  the  specific 
test  series.  There  are  two  types  of  shots: 
engineering  and  random.  Engineering 
shots  provide  information  and  data  to 
address  specific  vulnerability  or  lethality 
issues  for  a  specific  threat.  Random  shots 
are  selected  from  the  combat  distribution 
of  impact  conditions  (direction,  location, 
and  range)  for  the  threats  of  interest. 

The  minimum  number  of  engineering 
shots  should  be  selected  first  to  address 
the  vulnerability  and/or  lethality  critical 
issues.  Next,  the  number  of  random  shots 
required  for  each  threat  weapon  should  be 
selected.  Random  shots  should  be  reviewed 
to  determine  if  any  remaining  engineering 
shots  are  duplicated  or  if  a  critical  issue 
is  satisfied  by  a  random  shot.  Those 
remaining  engineering  shots  duplicated  by 
a  random  shot  should  be  eliminated. 

The  LFT&E  program  should  be  planned 
independent  of  constraints  and  then  efforts 
must  be  made  in  developing  and  approving 
the  strategy  to  obtain  relief  from  schedule 
and  resource  constraints.  The  most  likely 


outcome  of  this  process  is  compromise 
and  negotiating  strategies  that  meet  the 
spirit  and  intent  of  the  law  within  existing 
or  modified  constraints.  The  following 
parameters  must  be  selected  and  specified: 

♦  Posture  (offense  or  defense) 

♦  Range  (based  upon  offense  or  defense) 

♦  Angle  of  attack  (stratified  into  equal 
probability  intervals  to  ensure  sampling  over 
all  possible  attack  angles  with  small  sample 
sizes) 

♦  Target  side  (left  or  right) 

♦  Hull  or  turret 

♦  Horizontal  dispersion 

♦  Direction  of  horizontal  dispersion  (left  or 
right) 

♦  Vertical  dispersion 

♦  Direction  of  vertical  dispersion  (up  or  down) 

LFT&E  Reporting 

The  LFT&E  Report  documents  the  live 
fire  vulnerability/lethality  evaluation  and 
contains  the  assessment  of  the  critical 
issues  and  conclusions  concerning  the 
vulnerability/lethality  and  battlefield 
damage  assessment  and  system  repair 
(vulnerability  live  fire  programs  only). 

The  LFT&E  Report  addresses  the  test 
objectives,  issues,  and  criteria  as  defined  in 
the  LFSEP,  EDPs,  and  BDAR  Support 
Plan.  It  discusses  the  crosswalk  between 
results  and  the  evaluation  and  specifies 
any  limitations  relative  to  the  analysis.  The 
LFT&E  Report  objectively  addresses  all 
aspects  of  the  system  vulnerability/lethality 
based  on  the  likelihood  of  occurrence 
on  the  battlefield.  Not  all  vulnerabilities 
identified  in  Vulnerability  LFT&E 
can  be  fixed.  Constraints  on  system 
funding,  system  weight,  and  other  aspects 
necessitate  the  ranking  of  the  identified 
vulnerabilities  from  the  perspectives  of 
likelihood  of  occurrence  on  the  battlefield 
and  the  degree  of  system  degradation  given 
an  occurrence.  The  final  LFT&E  report 
provides  this  information  to  the  user  and  to 
the  PM  for  resolution. 
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Reliability ,  Availability,  and  Maintainability 


Reliability,  Availability,  and  Maintainability, 
collectively  known  as  RAM,  is  a  critical 
component  of  OS  and  is  inextricably 
linked  to  OE.  The  link  between  RAM 
and  effectiveness  and  the  individual 
components  of  RAM  can  be  summed  up  in 
the  following: 

♦  Weapon  systems  that  are  not  ready  for  use 
(available)  when  needed  prevent  effects 
from  occurring.  Weapon  systems  can  be 
unavailable  because  there  aren’t  enough  to 
go  around,  including  spares  to  keep  pace 
with  operational  demand;  or,  the  systems 
assigned  to  a  unit  are  undergoing  some 
maintenance  action  to  restore  or  preserve 
functionality. 

♦  A  weapon  system  that  malfunctions 
(reliability)  when  operating  affects  the 
Marine's  ability  to  achieve  a  desired  effect. 

A  weapon  system  that  malfunctions  requires 
repairs  to  either  correct  the  malfunction  or 
prevent  its  reoccurrence. 

♦  Systems  undergoing  repairs 
(maintainability)  are  unable  to  be  used 
when  called  for  at  any  random,  given 
point  in  time.  This  situation  is  exacerbated 
when  repair  actions  are  difficult  or  time 
consuming  causing  a  reduction  in  system 
availability. 

What  is  RAM?  To  understand  RAM  one 
must  first  understand  the  basic  definitions 
and  mathematical  expressions.  What 
follows  are  extracts  from  various  DOD 
publications. 

Availability  (A) 

Availability  is  defined  as  a  measure  of  the 
degree  to  which  an  item  is  in  an  operable 
and  committable  state  at  the  start  of  a 
mission  when  the  mission  is  called  for  at 
a  random  point  in  time.  Availability  is  the 
parameter  that  translates  system  Reliability 
and  Maintainability  characteristics  into  an 
index  of  effectiveness.  It  is  based  on  the 
question,  “Is  the  equipment  available  in  a 
working  condition  when  it  is  needed?” 


The  basic  mathematical  definition  of 
Availability  is  defined  by  the  equation: 

Availability  =  A 

Uptime 
Total  Time 

_  Uptime 
Uptime  +  Downtime 

Where 

Uptime  =  Time  the  system  is 
available  to  perform  designated 
mission. 

Downtime  =  Total  time  -  Uptime  = 
Time  system  is  unavailable  for  tasking. 

Actual  assessment  of  Availability  is 
accomplished  by  substituting  the  time 
based  elements  defined  above  into  various 
forms  of  this  basic  equation.  (DOD  1982) 

Material  Availability  (Am) 

Am  measures  the  percentage  of  systems  in 
operational  use — providing  a  meaningful 
snapshot  of  the  overall  efficiency  of  the 
program  elements  (design,  support  structure, 
use  profiles,  planned  and  unplanned 
maintenance  downtimes,  and  so  on)  to 
provide  the  necessary  capability  to  the 
warfighter  or  end  user.  A  is  not  a  substitute 
for  operational  readiness  metrics  (such 
as  Operational  Availability  (Ao),  Mission 
Reliability,  Mission  Capability  Rate).  Am 
provides  the  trade  space  between  acquisition 
and  support  costs  related  to  the  system 
design  and  support  approach.  Am  applies  to 
all  end  items  acquired  throughout  their  life 
cycle,  while  operational  readiness  metrics 
apply  to  end  items  in  the  operational 
environment  only — excluding  float/spare 
systems,  systems  at  depot  for  overhaul 
or  repair,  systems  that  have  not  been 
operationally  assigned,  and  so  on. 
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When  a  system  capability  that  includes 

planned  float/spare  systems  is  fielded,  Am  Uptime 

is  defined  by  the  following  equation:  °  Total  Time 

OT  +  ST 

A  Number  of  Operational  End  Items  ~  qj  +  ST  +  TPM  +  TCM  +  ALDT 

M  Total  Number  of  End  Items  Acquired  MTBM 

~  MTBM  +MDT 


Assessment  of  the  achieved  A  involves 

m 

determining  the  number  of  operational  end 
items  (i.e.,  those  ready  for  tasking)  divided 
by  the  total  number  of  end  items  acquired 
at  the  time  the  sample  is  taken. 

When  a  system  is  being  fielded  without 
float/ spares  then  all  acquired  end  items  are 
put  into  operational  service  and  remain 
there  unless  maintenance  is  required. 

Under  these  conditions,  the  following 
equation  is  used: 

A  Uptime 

Uptime  +  Downtime 
MTBM 

~  MTBM +  MDT 

Where 

MTBM  =  Mean  time  between 
maintenance  actions  requiring 
removal  of  system  from  operational 
use 

MDT  =  Average  system  downtime 
expected  given  the  anticipated 
support  structure.  (RAM-C 
Report  Manual  2009) 

Operational  Availability  (Ao) 

Ao  covers  all  segments  of  time  that  the 
equipment  is  intended  to  be  operational. 
Uptime  includes  operating  time  plus 
nonoperating  (stand-by)  time  (when  the 
equipment  is  assumed  to  be  operable). 
Downtime  includes  preventive  and 
corrective  maintenance  and  associated 
administrative  and  logistics  lead  time. 

All  are  measured  in  clock  time  using  the 
following  formula: 


Where 

Total  time  =  Uptime  +  Downtime 

OT  =  Operating  Time  =  when  the 
equipment  is  in  use,  further  defined  as 
the  time  during  the  accomplishment  of  a 
mission  profile  when  the  system  is  turned 
on  and  actively  performing  at  least  one,  if 
not  all,  of  its  functions. 

ST  =  Standby  Time  =  not  operating  but 
assumed  operable  in  a  specified  period, 
further  defined  as  Uptime  when  a  system  is 
not  committed  to  accomplishing  a  specific 
mission  profile. 

TPM  =  Total  Preventive  Maintenance 
Time  =  scheduled  maintenance  time  per 
specified  period. 

TCM  =  Total  Corrective  Maintenance 
Time  =  unscheduled  maintenance  time  per 
specified  period. 

ALDT  =  Administrative  and  Logistics 
Down  Time  =  time  spent  waiting  for  parts, 
administrative  processing,  maintenance 
personnel,  or  transportation  per  specified 
period. 

This  relationship  is  intended  to  provide  a 
realistic  measure  of  equipment  availability 
when  the  equipment  is  deployed  and 
functioning  in  a  combat  environment. 

Ao  is  used  to  support  operational  testing 
assessment,  life  cycle  costing,  and  force 
development  exercises.  One  significant 
problem  associated  with  determining  Ao  is 
that  it  becomes  costly  and  time-consuming 
to  define  the  various  parameters. 

Defining  ALDT  and  TPM  under  combat 
conditions  is  not  feasible  in  most  instances. 
Nevertheless,  the  Ao  expression  does 
provide  an  accepted  technique  of  relating 
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standard  reliability  and  maintainability 
elements  into  an  effectiveness-oriented 
parameter  (DOD  1982). 

Achieved  Availability  (Aa) 

A  is  frequently  used  during  development 
testing  and  initial  production  testing  when 
the  system  is  not  operating  in  its  intended 
support  environment.  Excluded  are  operator 
before-and-after  maintenance  checks  and 
standby,  supply,  and  administrative  waiting 
periods.  Aa  is  much  more  a  system  hardware 
oriented  measure  than  is  A  ,  which 
considers  operating  environment  factors. 

It  is,  however,  dependent  on  the  preventive 
maintenance  policy,  which  is  greatly 
influenced  by  non-hardware  considerations. 
All  times  are  measured  in  clock  time  using 
the  formula: 


a  OT  +  TCM  +  TPM 

Inherent  Availability  (A.) 

A  is  useful  in  determining  basic  system 
operational  characteristics  under  conditions 
which  might  include  testing  in  a 
contractor’s  facility  or  other  controlled  test 
environment.  Likewise,  A  becomes  a  useful 
term  to  describe  combined  Reliability 
and  Maintainability  characteristics  or  to 
define  one  in  terms  of  the  other  during 
early  conceptual  phases  of  a  program 
when,  generally,  these  terms  cannot  be 
defined  individually.  Since  this  definition 
of  Availability  is  easily  measured,  it  is 
frequently  used  as  a  contract-specified 
requirement. 

A.  defines  system  availability  with  respect 
only  to  Operating  Time  and  Corrective 
Maintenance  and  can  be  expressed  using 
the  formula: 

_  OT  _  MTBF 
'  ”  OT  +  TCM  ~  MTBF  +  MTTR 


Where 

MTBF  =  Mean  Time  Between  Failures=  the 
average  time  during  which  all  parts  of  the 
item  perform  within  their  specified  limits, 
during  a  particular  measurement  period 
under  stated  conditions  (DOD  2005). 

MTTR  =  Mean  Time  To  Repair  = 
includes  diagnostic  time  (time  to  detect 
and  isolate  failure);  time  to  repair  (in-place 
repair  or  removal  and  replacement);  and 
time  required  to  validate  the  repair  (e.g., 
functional  check)  (DOD  2005). 

Under  these  idealized  conditions  Standby 
and  Delay  Times  associated  with 
Scheduled  or  Preventive  Maintenance  can 
be  ignored  as  well  as  Administrative  and 
Logistics  Down  Time.  As  is  evident  from 
this  definition,  A  provides  a  very  poor 
estimate  of  true  combat  potential  for  most 
systems  because  it  provides  no  indication 
of  the  time  required  to  obtain  required  field 
support.  This  term  should  normally  not  be 
used  to  support  an  operational  assessment 
(DOD  1982). 

Reliability 

Reliability  measures  the  probability  that 
the  system  will  perform  without  failure 
over  a  specified  interval  under  specified 
conditions.  Reliability  must  be  sufficient  to 
support  the  warfighting  capability  needed 
in  its  expected  operating  environment. 

Considerations  of  Reliability  must  support 
both  Aq  and  A  .  Reliability  may  be  expressed 
initially  as  a  desired  failure-free  interval  that 
can  be  converted  to  a  failure  frequency  for  use 
as  a  requirement  (DOD  2009). 

Two  very  different  system  Reliability 
design  objectives  exist.  One  is  to 
enhance  system  effectiveness;  the  other 
is  to  minimize  the  burden  of  owning  and 
operating  the  system.  The  first  objective 
is  addressed  by  means  of  Mission 
Reliability,  the  second  by  means  of 
Material  or  Logistics-related  Reliability. 
Measures  of  Mission  Reliability  address 
only  those  incidents  that  affect  mission 
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accomplishment.  Measures  of  Logistics- 
related  Reliability  address  all  incidents  that 
require  a  response  from  the  logistics  system 
(DOD  1982). 

Mission  Reliability 

Mission  Reliability  is  the  probability  that 
a  system  will  perform  mission  essential 
functions  for  a  period  of  time  under  the 
conditions  stated  in  the  mission  profile. 
Mission  Reliability  for  a  single  shot  type 
of  system,  i.e.,  a  pyrotechnic  device,  would 
not  include  a  time  period  constraint.  A 
system  with  high  Mission  Reliability  has  a 
high  probability  of  successfully  completing 
the  defined  mission.  A  typical  Mission 
Reliability  metric  is  Mean  Time  Between 
Operational  Mission  Failure  (MTBOMF) 
for  systems  with  a  continuous  Reliability 
requirement.  MTBOMF  is  defined  as  a 
measure  of  operational  mission  Reliability 
for  the  system;  it  is  the  average  time 
between  operational  mission  failures  that 
cause  a  loss  of  the  system’s  “mission”  as 
defined  by  the  customer.  This  parameter 
may  include  both  hardware  and  software 
“failures. ’’This  parameter  also  includes 
failures  that  are  generally  attributed 
to  human  errors  during  operation  and 
maintenance  that  cause  failures  (DOD 
2005).  MTBOMF  can  be  expressed  by  the 
equation  (MOA  on  MOT&E  2009): 

TOT 

MTBOMF  = - 

#  of  OMFs 

Where 

TOT  =  Total  Operating  Time  = 
summation  of  all  periods  of  Operating 
Time  when  the  equipment  is  in  use. 
Operating  Time  is  further  defined  as  the 
time  during  the  accomplishment  of  a 
mission  profile  when  the  system  is  turned 
on  and  actively  performing  at  least  one,  if 
not  all,  of  its  functions. 

OMFs  =  Operational  Mission  Failures  = 
any  incident  or  malfunction  of  the  system 
that  causes  (or  could  cause)  the  inability  to 
perform  one  or  more  designated  mission 


essential  functions  (TRADOC/AMC 
1987). 

Therefore,  a  Mission  Reliability  analysis 
must  include  the  definition  of  mission 
essential  functions  (DOD  1982). 

Mission  Essential  Functions 

Mission  Essential  Functions  (mef)  are 
the  minimum  operational  tasks  that  the 
system  must  be  capable  of  performing 
to  accomplish  the  assigned  mission. 
Descriptions  of  mission  essential  functions 
should  be  in  operational  terms  that  relate 
to  mission  requirements.  The  equipment 
operator  should  be  able  to  readily  identify 
the  loss  of  a  mission  essential  function 
(DOD  1982). 

Mefs  have  both  a  qualitative  and 
quantitative  aspect.  Qualitatively  mefs 
are  brief  statements,  usually  infinitives, 
that  declare  why  the  given  equipment 
is  needed,  what  its  purpose  is.  Typical 
mefs  include  “to  move,”  “to  shoot,”  and 
“to  communicate.”  Quantitative  mefs  are 
followed  up  with  quantitative  information 
to  describe  the  break  point  between 
satisfactory  and  unsatisfactory  performance 
of  the  function.  Using  the  “to  move” 
qualitative  example  the  quantitative  aspects 
might  be  to  “travel  at  30  mph  on  cross¬ 
country  under  full  load,”  (TRADOC/ 
AMC  1987). 

Material  Reliability 

Material  Reliability  (Rm),  also  known  as 
Logistics  Reliability  or  Basic  Reliability, 
is  a  characteristic  of  the  final  system 
design.  All  indicated  and  recorded  failures, 
even  those  that  do  not  affect  successful 
completion  of  the  mission,  eventually 
result  in  some  corrective  action  (repair). 
Corrective  action  often  includes  some 
level  of  repair  or  inspection  to  mitigate  the 
failure.  Repairs  in  this  case  can  consist  of 
removal  and  replacement,  in-place  repair, 
or  some  combination  thereof  for  the  failed 
item  (DOD  2005). 
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R  is  defined  by  the  Mean  Time  Between 

m  J 

Failures  (MTBF  )  of  the  system: 

MTBF  =  — 
k 

Where 

^  =System  failure  rate=the  total  number 
of  failures  within  an  item  population, 
divided  by  the  total  time  expended 
by  that  population,  during  a  particular 
measurement  interval  under  stated 
conditions. 

Logistics-related  Reliability  measures,  as 
indicated  above,  must  be  selected  so  that 
they  account  for  or  address  all  incidents 
that  require  a  response  from  the  logistics 
system.  Logistics-related  Reliability  may 
be  further  subdivided  into  Maintenance- 
related  Reliability  and  Supply-related 
Reliability.  These  parameters  respectively 
represent  the  probability  that  no  Corrective 
Maintenance  or  the  probability  that  no 
unscheduled  supply  demand  will  occur 
following  the  completion  of  a  specific 
mission  profile  (DOD  1982). 

Maintainability 

Maintainability,  along  with  Reliability,  is 
one  of  the  two  major  system  characteristics 
that  combine  to  form  Availability. 
Maintainability  and  maintenance  are 
not  the  same.  Maintainability  is  a  design 
consideration  whereas  maintenance  is 
the  consequence  of  the  design  (DOD 
1982).  Maintainability  is  defined  as  the 
probability  that  an  item  can  be  retained 
in,  or  restored  to,  a  specified  condition  in  a 
given  time  when  maintenance  is  performed 
by  personnel  having  specified  skill  levels, 
using  prescribed  procedures  and  resources, 
at  each  prescribed  level  of  maintenance  and 
repair.  Maintainability  is  a  function  of  the 
design  (DOD  2005). 

Maintenance 

Maintenance  is  the  term  used  to  define 
all  actions  required  to  retain  an  item  in, 
or  restore  it  to,  a  specified  condition.  This 


includes  diagnosis,  repair  and  inspection. 
Maintenance  can  be  further  subdivided  into 
Preventive  and  Corrective  Maintenance. 

Preventive  (Scheduled)  Maintenance 

Preventive  Maintenance  (PM)  is  defined 
as  systematic  inspection,  detection,  and 
correction  of  incipient  failures  either  before 
they  occur  or  before  they  develop  into 
major  defects.  Adjustment,  lubrication, 
and  scheduled  checks  are  included  in 
the  definition  of  preventive  maintenance 
(DOD  1982). 

Preventive  Maintenance  actions  are 
considered  Scheduled  Maintenance  Actions 
(SMA).  SMAs  are  services  or  repairs 
performed  at  intervals  measured  by  calendar 
time,  by  use  (hours  of  operations,  rounds 
fired,  etc.),  or  by  condition  (wear  limits,  low 
battery  power,  depleted  lubrication,  etc.). To 
qualify  as  an  SMA  the  maintenance  must 
be  prescribed  by  an  equipment  publication 
and  enough  latitude  in  the  time  to  perform 
the  maintenance  must  exist  that  it  can  be 
done  in  a  slack  period  between  missions 
(TRADOC/AMC  1987). 

A  system  undergoing  Preventive 
Maintenance  can  be  considered  either 
available  or  unavailable  depending  on 
the  effect  of  the  maintenance  action 
on  the  system’s  ability  to  perform  mefs. 
Preventive  Maintenance  that  inhibits  the 
accomplishment  of  a  mef  causes  the  system 
to  be  unavailable.  An  example  would  be 
a  routine  brake  inspection  on  a  vehicle 
whose  mef  is  to  move.  If  the  wheel  of  the 
vehicle  has  to  be  removed  to  inspect  the 
brakes  during  a  Preventive  Maintenance 
check,  then  that  vehicle  loses  the  ability  to 
perform  the  move  mef.  Therefore,  during 
the  Preventive  Maintenance  period  the 
system  is  considered  unavailable  until  such 
time  as  the  vehicle  is  capable  of  performing 
all  of  its  mefs. 

Corrective  (Unscheduled) 
Maintenance 

Corrective  Maintenance  (CM)  is  defined 
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as  that  maintenance  performed  on  a  non- 
scheduled  basis  to  restore  equipment  to 
satisfactory  condition  by  correcting  a 
malfunction.  All  CM  actions  are  considered 
unscheduled  maintenance  actions.  The 
importance  of  the  repair  action  will  often 
dictate  when  the  repair  takes  place.  Repair 
actions  can  be  performed  immediately, 
deferred  until  after  the  mission  but  before 
the  next  mission,  or  deferred  until  a  time 
when  the  system  is  not  required  to  be 
available.  Deferring  maintenance  until 
a  scheduled  period  of  Down  Time  does 
not  change  the  corrective  action  from 
unscheduled  to  scheduled.  The  existence  of 
a  failure  or  malfunction  is  what  determines 
whether  or  not  the  maintenance  action  is 
unscheduled  vice  scheduled. 

Depending  on  the  urgency  of  the  repair 
actions  and  the  impact  to  mefs,  CM 
actions  can  be  classified  in  three  basic 
categories:  Operational  Mission  Failures 
(OMF),  Essential  Maintenance  Actions 
(EMA),  and  Unscheduled  Maintenance 
Actions  (UMA). 

Operational  Mission  Failures 

Operational  Mission  Failures  (OMF)  are 
incidents  that  require  immediate  resolution 
in  order  to  resume  or  perform  a  mission. 
When  a  system  is  undergoing  corrective 
action  as  a  result  of  an  OMF  the  system  is 
considered  unavailable.  If  the  malfunction 
is  such  that  the  repair  can  be  deferred, 
and  the  mission  can  be  continued,  then 
the  incident  is  not  considered  an  OMF. 

A  special  case  of  OMFs  is  called  crew- 
correctable  maintenance  actions  (COMA) 
(TRADOC/AMC  1987). 

Crew-Correctable  Maintenance  Action 

CCMAs  are  optional,  but  when  used  they 
are  defined  as  those  minor  interruptions  of 
the  mission  which  the  crew  overcomes  by 
quick,  local  action.  CCMAs  are  resolved  by 
the  crew  using  only  the  system’s  onboard 
tools,  repair  parts,  and  spares.  Crew  action 
need  not  be  maintenance,  but  can  be 
simply  a  powering  down  and  powering 


up  of  the  equipment.  The  amount  of  time 
allowed  to  a  CCMA  before  the  incident 
becomes  a  more  serious  stoppage  (i.e.,  an 
OMF)  depends  on  the  mission.  Within  a 
given  system  different  CCMA  times  may  be 
allowed  for  the  different  mefs  according  to 
the  function  and  its  urgency  to  the  mission. 

CCMAs  may  occur  multiple  times  in 
a  mission.  The  occurrence  of  multiple 
CCMAs  in  a  single  mission  may  have  the 
aggregate  effect  comparable  to  an  OMF. 

In  other  words,  when  too  many  minor 
interruptions  occur,  the  net  effect  can  be  a 
failure  of  the  mission.  The  cumulative  effect 
should  be  defined  in  advance  in  the  FD/ 

SC  Charter  (TRADOC/AMC  1987). 

Essential  Maintenance  Actions 

EMAs  are  incidents  in  which  the 
malfunction,  or  the  deviation  from 
specification,  has  to  be  corrected  for 
complete  mission  readiness.  At  times 
special  conditions  exist  or  alternative 
methods  or  components  for  carrying  out 
a  mission  are  present  that  make  what  is 
otherwise  considered  an  OMF  to  be  an 
EMA.  As  an  example,  if  the  headlights  are 
broken  during  daylight-only  missions,  then 
the  incidents  are  considered  EMAs  vice 
OMFs  (TRADOC/AMC  1987). 

Unscheduled  Maintenance  Actions 

All  maintenance  actions  not  otherwise  the 
result  of  OMFs,  CCMAs,  or  EMAs  are 
considered  UMAs. 

Maintenance  Metrics 

Maintenance  metrics  are  based  on  a  few 
key  observations  that  are  predominately 
based  on  time,  personnel,  and  parts/spares 
used.  From  this  information  Mean  Time 
To  Repair  (MTTR),  Maximum  Time 
To  Repair  (MaxTTR),  and  Maintenance 
Ratio  (MR)  can  be  derived.  The  other 
component  of  maintenance  relates  to  the 
logistics  aspect,  and  is  Administrative  and 
Logistics  Down  Time. 
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Mean  Time  to  Repair 

MTTR,  also  called  Mean  Corrective 
Maintenance  Time,  is  the  total  Corrective 
Maintenance  Down  Time  accumulated 
during  a  period  divided  by  the  total 
number  of  Corrective  Maintenance 
Actions  completed  during  the  same  period. 
MTTR  includes 

♦  Diagnostic  Time  (time  to  detect  and  isolate 
failure) 

♦  Time  to  repair  (in-place  repair  or  removal 
and  replacement  of  the  failed  item) 

♦  Time  required  to  validate  the  repair  (e.g., 
functional  check)  (DOD  2005) 

MTTR  is  commonly  used  as  an  on- 
equipment  measure  but  can  be  applied  to 
each  maintenance  level  individually  MTTR 
is  expressed  by  the  following  formula: 


#  CM  Actions 

MTTR  does  not  account  for  frequency 
of  corrective  maintenance  items  or  for 
the  number  of  man-hours  expended; 
therefore,  MTTR  is  not  a  good  measure 
of  maintenance  burden  (DOD  1982). 

An  appropriate  measure  for  maintenance 
burden  is  Maintenance  Ratio  (MR). 

Maximum  Time  to  Repair 

MaxTTR  is  the  maximum  Corrective 
Maintenance  Down  Time  within  which 
either  90  or  95  percent  (as  specified)  of  all 
Corrective  Maintenance  Actions  can  be 
accomplished.  MaxTTR  is  useful  in  special 
cases  where  the  system  has  a  tolerable 
Down  Time.  An  absolute  maximum  would 
be  ideal  but  is  impractical  because  some 
failures  will  inevitably  require  exceptionally 
long  repair  times  (DOD  1982). 

Maintenance  Ratio 

MR  is  the  cumulative  number  of  man-hours 
of  maintenance  expended  in  direct  labor 
during  a  given  period  of  time,  divided  by  the 
cumulative  number  of  end-item  operating 
hours  (or  rounds  or  miles)  during  the  same 


time.  MR  is  expressed  with  the  formula: 

maintenance  man  hours 

MR  =  - - - 

operating  hours 

The  MR  is  expressed  at  each  level  of 
maintenance  and  summarized  for  all 
levels  of  maintenance  combined.  Both 
Corrective  and  Preventive  Maintenance  are 
included.  Man-hours  for  off-system  repair 
of  replaced  components  and  man-hours 
for  daily  operational  checks  are  included 
for  some  classes  of  systems.  MR  is  a  useful 
measure  of  the  relative  maintenance  burden 
associated  with  a  system.  It  provides  a 
means  of  comparing  systems  and  is  useful 
in  determining  the  compatibility  of  a 
system  with  the  size  of  the  maintenance 
organization  (DOD  1982). 

Administrative  and 
Logistics  Down  Time 

Administrative  and  Logistics  Down 
Time  (ALDT)  is  the  time  spent  waiting 
for  parts,  administrative  processing, 
maintenance  personnel,  or  transportation 
per  specified  period  (DOD  1982).  During 
ALDT  active  maintenance  is  not  being 
performed  on  the  downed  piece  of 
equipment  (TRADOC/AMC  1987). 

Linking  Metrics  to 
Time-Based  Models 

The  Measures  of  Suitability  previously 
presented  are  inextricably  linked  to 
Operational  Mode  Summary  and 
Mission  Profile  for  the  system  and  time 
categorizations. 

All  systems  have  some  form  of  time 
characterizations,  which  identify  the  state  a 
system  is  in  at  any  given  time.  For  example, 
spare  systems  sitting  in  a  warehouse  may  be 
in  Inactive  Time,  while  systems  assigned  to 
operational  units  would  be  in  Active  Time. 
An  individual  system  may  be  employed 
on  a  specific  mission  profile  putting  it  in 
Mission  Time,  while  another  might  be  in 
between  missions  and  thus  currently  in 
Standby  Time. 
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Time 


Many  of  the  times  categories 
have  already  been  identified  as 
part  of  the  RAM  metrics  such 
as  Operating  Time,  Corrective 
and  Preventive  Maintenance 
Down  Times,  Administrative 
and  Logistics  Down  Time, 

Standby  Time,  etc.  Figure 
6-2-1  depicts  the  discrete  time 
categories  that  can  be  used 
to  classify  time  for  a  system 
(TRADOC/AMC  1987). 

A  system  can  be  in  only 
one  time  category  at  a  time, 
although  some  exceptions  exist 
that  the  figure  does  not  depict. 

The  time  categories  are  used 
to  identify  the  state  of  any 
given  system  on  a  timeline  as 
depicted  in  figure  6-2-2  (next 
page). 

The  timeline  illustrates  a 
single-system,  single-mission 
example.  In  the  example,  the 
system  starts  the  timeline  in 
standby  time  and  remains  in 
that  time  status  until  such 
time  as  a  mission  are  called 
for  at  the  random  given  point 
in  time.  Mission  time  for 
this  system  begins  with  pre¬ 
operations  checks,  although 
it’s  worth  noting  that  for  this 
fictional  system  post-operations  checks 
are  not  considered  part  of  the  mission. 
Following  pre-operations  checks  the 
system  is  placed  in  alert  time,  a  special 
case  of  standby  time  where  the  system 
is  committed  to  a  specific  mission,  is 
considered  operable,  but  is  not  currently 
operating.  When  the  operators  are  given 
the  command  to  begin  operating  the 
system  the  time  categorization  changes 
from  alert  time  to  operating  time.  During 
operating  time  at  least  one  or  more  mefs 
are  in  use. 

This  example  further  illustrates  two  types 
of  unscheduled  maintenance,  the  arrival 


Inactive  Time 


Active  Time 


of  each  occurs  during  operating  time. 

In  this  example  the  arrival  of  the  failure 
causes  the  loss  of  a  mef.  Upon  arrival  of 
the  failure,  the  crew  immediately  begins 
crew  correctable  maintenance  actions  and 
restores  the  necessary  functionality,  thus 
preventing  the  mission  from  becoming  a 
complete  failure.  The  second  failure  that 
arrives  does  not  cause  the  loss  of  a  mef 
because  a  redundancy  exists  that  prevents 
the  mission  from  being  a  complete  failure; 
therefore,  the  crew  takes  no  immediate 
action  to  repair/restore  the  loss  due  to  the 
failure.  Ultimately  the  maintenance  action  is 
deferred  until  the  completion  of  the  mission. 


Fig.  6-2-1.  Time 

Classification 

Dendrite 
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Fig.  6-2-2. 
Sample  System 
Timeline 


Failure  -  Loss  of 
Mission  Essential 
Function 
(Non-Deferrable) 


Failure  -  Non- 
Mission  Essential 
Function 
(Deferrable) 


Essential 

->•  Maintenance  Action 
(EMA) 


Crew  Correctible 
Maintenance  Action 
(CCMA)  * 


*  Fixed  by  the  crew  using  onboard  tools,  equipment,  and  spares  within  the  specified  timelimit. 


In  this  example,  the  failure  is  considered 
an  EMA.  Therefore,  the  restoration  of  the 
functionality  must  take  place  prior  to  the 
start  of  the  next  mission.  The  time  spent 
restoring  the  functionality  is  considered  as 
part  of  downtime  under  CM. 

Figure  6-2-3  (next  page)  illustrates  the 
links  between  time  categories  and  RAM 


metrics.  As  illustrated  in  the  figure  specific 
time  categories  like  operating  time, 
standby  time,  alert  time,  and  downtimes 
associated  with  preventive  and  corrective 
maintenance,  administration,  and  logistics 
feed  directly  into  the  equations. 
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*  Alert  Time  is  a  special  case  of  Standby  Time.  In  Alert  Time  a  system  is  committed  to  a 
specific  mission,  considered  operable,  but  not  actually  operating  during  that  time  period. 


Links  Between  Time 
Categories  and  RAM 
Metrics 


Operational  Mode  Summary 
and  Mission  Profile 

The  combat  developer  must  articulate  the  mix 
of  ways  the  system  performs  its  operational 
role  in  an  Operational  Mode  Summary  and 
Mission  Profile  (OMS/MP).  An  integral 
part  of  the  analysis  is  the  determination 
of  the  frequency  of  task  performance,  the 
conditions  under  which  they  are  performed, 
and  the  standards  which  constitute  acceptable 
performance.  This  description  of  tasks, 
frequency,  conditions,  and  standards  forms 
the  basis  for  the  OMS/MP. 


Operational  Mode  Summary 

The  OMS  describes  the  relative  frequency 
of  the  various  missions,  which  systems  will 
be  involved  in  those  missions,  and  the  types 
of  environmental  conditions  to  which  the 
system  will  be  exposed  during  the  system 
life  cycle  (DOD  2009).  The  contents  to 
look  for  in  an  OMS  are  as  follows: 

♦  General  statement  of  broad  missions  that 
the  equipment  will  be  expected  to  perform 
on  the  battlefield. 

♦  Separately  addresses  both  wartime  and 
peacetime  use. 
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♦  Addresses  special  conditions  of  use,  such  as 
high-intensity  wartime  usage. 

♦  Expected  number  of  occurrences,  operating 
time,  and  calendar  time  of  each  mission  or 
the  percentage  of  the  systems  involved  in 
each  mission. 

♦  Expected  breakdown  of  environmental 
conditions  in  which  the  entire  fleet  of 
systems  is  expected  to  be  used. 

♦  States  the  total  Operating  and  Alert  Time 
associated  with  each  mission  (i.e.,  time  that 
the  system  is  required  to  be  operable  and 
committed  on  a  specific  mission,  even  if  not 
operating)  (TRADOC/AMC  1987). 

When  elements  of  the  OMS  are  not 
present  a  clarification  to  the  combat 
developer  should  be  initiated  to  address  the 
lack  of  information. 

Tables  6-2-1  and  6-2-2  illustrate  a 
fictitious  example  of  a  typical  wartime 
OMS  for  the  XYZ  system.  The  OMS 
lists  the  types  of  missions  and  number  of 


missions.  In  addition,  the  OMS  lists  the 
quantities  of  operating,  alert,  and  clock 
time  associated  with  each  mission  type.  The 
more  detailed  breakdown  of  a  mission  can 
be  found  in  the  Mission  Profile.  The  OMS 
also  describes  the  operating  envelope  in 
terms  of  environment  for  the  system. 

Mission  Profile 

The  Mission  Profile  describes  the  tasks, 
events,  durations,  frequency,  operating 
conditions,  and  environment  of  the  system 
for  each  phase  of  a  mission  (DOD  2009). 
The  Mission  Profile  also  defines  a  time- 
phased  description  of  the  operational  events 
and  environments  an  item  experiences 
from  beginning  to  end  of  a  specific  mission. 
(TRADOC/AMC  1987).  The  contents  to 
look  for  in  an  MP  are  as  follows: 

♦  Profiles  should  be  based  on  typical  scenario 
for  the  system. 

♦  State  specific  amounts  of  operation  (e.g., 


Table  6-2-1 .  Fictitious  OMS  for  the  XYZ  System 


XYZ 

Missions 

Operating 

Time 

(a) 

OT+Alert 

Time 

(b) 

Calendar 

Time 

(o) 

No.  of 
Missions 
(d) 

Total  OT 
(a) x (d) = 
(e) 

Total  OT+AT 
(b)  x  (d)  =  (f) 

Total  CT 
(c)  x  (d)  =  (g) 

Covering 

Force 

16  hr 

16  hr 

18  hr 

2 

32  hr 

32  hr 

36  hr 

Forward 
Line  of 
Troops 
Defense  * 

68  hr 

72  hr 

72  hr 

10 

680  hr 

720  hr 

720  hr 

Deep 

Strike 

16  hi* 

20  hr 

20  lv 

1 

16  hr 

20  hr 

20  hr 

Counter 

Attack 

25  hr 

30  hr 

30  hr 

2 

50  hr 

60  hr 

60  hr 

Total 

N/A 

N/A 

N/A 

15 

778  hr 

832  hr 

832  hr 

*Detailed  breakdown  can  be  found  in  the  example  for  Mission  Profile. 


Table  6-2-2.  Fictitious  Environmental  OMS  for  the  XYZ  System 


Climate 

Environment 

-  Temperature:  Hot  20%,  Basic  60%,  Cold  15%,  Severe  5% 

-  Humidity  Range:  15%  -  95% 

-  Movement  Terrain:  Primary  10%,  Secondary  35%,  X-Country  55% 

Weather 

Environment 

-  Precipitation  Type:  Rain,  Light  Snow 

Terrain 

Conditions 

-  Soil:  Clay,  Loam,  Sand 

-  Vegetation:  Coniferous  Forest 

-  Slope:  0%  to  10%  (over  50%  of  area) 
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hours,  rounds,  miles,  and/or  cycles)  for  each 
mef  within  the  mission. 

♦  Should  be  consistent  with  future  doctrine 
and  tactics. 

♦  Information  should  be  provided  on  a 
timeline,  a  summarization,  or  other  type  of 
format. 

♦  Environmental  conditions  for  each  mission. 

When  elements  of  the  MP  are  not  present 
a  clarification  to  the  combat  developer 
should  be  initiated  to  address  the  lack  of 
information.  Mission  Profiles  are  related 
to  the  Operational  Task  Analysis  (OTA) 
previously  mentioned  in  other  sections  of 
the  manual.  When  Mission  Profiles  exist 
for  a  system  they  should  be  used  as  the 
basis  for  the  OTA. 

Tables  6-2-3  and  6-2-4  illustrate  a  fictitious 
example  of  a  typical  wartime  mission 
profile  for  the  XYZ  system’s  defensive 
mission.  The  example  illustrates  the  type 


of  information  needed  for  a  thorough 
understanding  of  the  mission  profile.  Table 
6-2-3  identifies  the  tasks  to  be  performed 
during  Operating  Time,  including  the 
number  of  occurrences,  time  allotted,  and 
the  cumulative  time.  Table  6-2-4  identifies 
the  expected  environmental  conditions. 
Information  resources  for  populating  the 
environmental  table  are  found  in  the  MIL- 
STD-810  series  and  the  Universal  Naval 
Task  List  (MCO  3500.26  series).  The 
details  of  each  table  are  complementary  but 
not  identical.  For  example,  the  movement 
terrain  is  not  the  same  as  the  OMS  and  the 
MP.  The  difference  can  be  explained  in  that 
the  OMS  is  an  aggregate  of  all  mission 
profiles,  each  of  which  varies  in  movement. 

Linking  Time-Based  Models  to 
Failure  Definition/Scoring  Criteria 

Following  the  development  of  the 
CONOPS  and  OMS/MP,  the  combat 


Table  6-2-3.  XYZ  System  Defensive  Mission  Profile 


XYZ  Defensive  Mission 
Tasks 

Number  of 
Occurrences 

Operating  Time  for 
Each  Task 

Total  Operating  Time 

Movement 

12 

30  min 

6.0  hr 

Set-up  and  Pre-Ops  Checks 

12 

20  min 

4.0  hr 

Search  and  Surveillance 

80 

30  min 

40.0  hr 

Target  Acquisition 

36 

15  min 

9.0  hr 

Track 

24 

5  min 

2.0  hr 

Fire  (Air) 

9 

200  Rounds  at  100  rds/ 
min  (2  min) 

0.3  hr 

Fire  (Ground) 

28 

400  Rounds  at  50  rds/ 
min  (8  min) 

3.7  hr 

Tear  Down 

12 

15  min 

3.0  hr 

Total  * 

N/A 

N/A 

68.0  hr 

*For  the  mission,  all  time  that  the  system  is  not  operating  is  required  as  alert  time. 

Table  6-2-4.  XYZ  System  Defensive  Profile  (Environmental) 


Climate 

Environment 

Weather 

Environment 

-  Temperature:  Hot  20%,  Basic  60%,  Cold  15%,  Severe  5% 

-  Humidity  Range:  15%  -  95% 

-  Movement  Terrain:  Primary  20%,  Secondary  30%,  X-Country  50% 

-  Precipitation  Type:  Rain,  Light  Snow 

Terrain 

Conditions 

-  Soil:  Clay,  Loam,  Sand 

-  Vegetation:  Coniferous  Forest 

-  Slope:  0%  to  10% 
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developer  must  decide  what  minimal 
operational  tasks  the  system  must  be  able 
to  perform  in  order  to  accomplish  its 
mission,  as  well  as  what  the  associated 
mefs  are  in  order  to  identify  and  classify 
potential  failures.  This  information  is 
documented  in  the  FD/SC.  Refer  to 
chapter  3-2  for  information  about  tailoring 
the  FD/SC  Charter. 

Incident  Classification 

The  first  step  in  the  process  for  a  given 
test  incident  is  to  determine  classification. 
In  the  classification  step,  the  members 
of  the  scoring  conference  determine  if 
an  incident  is  related  to  Reliability  or 
Maintainability  of  the  equipment  as  it 
will  be  expected  to  be  used  in  the  field 
environment.  See  figure  6-1-4,  Test 
Incident  Scoring  Flowchart,  at  the  end  of 
this  chapter. 

No  Test 

Incidents  that  are  judged  not  pertinent  to 
RAM  parameters  are  classified  as  No  Test. 
Incidents  classified  as  No  Test  include: 

♦  Pre-test  Checkout.  Any  incident  observed 
during  the  designated  burn-in,  pre-test 
inspection,  or  other  pre-test  activity  is 
classified  as  No  Test.  The  Test  Plan  must 
specify  the  length  of  the  burn-in  period 
(the  number  of  miles,  rounds,  or  hours)  to 
permit  a  determination  of  when  the  pre-test 
period  has  ended. 

♦  Equipment  Modification.  Maintenance 
done  to  install  a  hardware  kit  or  to 
incorporate  a  redesigned  component  is 
classified  as  No  Test.  However,  if  the 
replaced  part  was  not  functioning  when  it 
was  being  replaced,  that  malfunction  will 
be  scored  on  its  own  merits.  A  subsequent 
malfunction  of  the  installed  part  will  also  be 
scored  on  its  own  merits. 

♦  Test-Peculiar  Incident.  An  incident 
caused  by  someone  not  acting  as  a  test 
player  (crew  member  or  maintainer),  or  by 
equipment  not  part  of  the  system  being 
tested  is  classified  as  No  Test.  An  example 
of  this  is  an  engineering  evaluation  and 
the  maintenance  done  in  furtherance 


of  that  evaluation.  This  classification 
also  includes  malfunctions  to  or  caused 
by  test  instrumentation.  However,  an 
incident  caused  by  test-peculiar  equipment 
(equipment  used  in  the  test  in  lieu  of  the 
equipment  to  be  fielded)  will  be  scored 
under  its  own  merits  because  if  the  test 
planners  have  introduced  equipment  for  the 
purposes  of  the  test,  they  have  judged  it  to 
be  an  adequate  substitute  for  the  equipment 
to  be  fielded;  hence  its  failures  are  to  be 
regarded  as  representative  of  the  failures  of 
the  equipment  to  be  fielded. 

♦  Daily  Checks  and  Services.  These  are  checks 
and  services,  performed  by  the  operator 

(or  by  the  crew,  if  applicable)  using  only 
repair  parts  and  On-Equipment  Material 
(OEM)  in  accordance  with  the  equipment 
publication  before,  during,  or  after  the 
operation  of  the  equipment.  Checks  and 
services  that  meet  these  conditions  are 
classified  as  No  Test. 

♦  Test-Directed  Abuse.  An  incident  in  which 
the  tester  directs  the  deliberate  abuse  of 
the  system  (e.g.,  a  test  to  over-stress  the 
performance  limits  of  the  system),  whether 
called  for  by  the  test  plan  or  not.  However, 
damage  to  the  system  willfully  caused  by 
the  operator  or  maintainer  and  not  directer 
by  the  tester  will  be  scored  under  its  own 
merits. 

♦  Non-RAM  Oriented.  This  is  a  catch-all 
term  to  capture  those  incidents  in  which  a 
TIR  has  been  prepared,  but  which  have  no 
bearing  on  the  RAM  assessment.  Example 
are  suggested  improvements;  reports  on 
inadequate  test  procedures;  reports  on 
unacceptable  replacement  parts,  provided 
they  were  discovered  before  or  during 
installation;  reports  on  the  equipment’s 
consistent  inability  to  meet  performance 
specifications  even  though  no  actual 
malfunction  has  occurred;  suggested  human 
factors  improvements;  and  recommended 
changes  to  the  system  support  package. 

Crew  Correctable  Maintenance 
Action  (Optional) 

The  second  step  in  the  classification 
process  is  to  determine  if  the  incident 
was  crew  correctable.  If  the  incident  was 
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a  malfunction  that  was  correctable  by 
the  crew,  within  the  specified  time  limits, 
using  only  the  system's  onboard  tools, 
repair  parts,  and  spares,  then  the  incident 
should  be  scored  as  a  CCMA. 

♦  If  the  system  and  its  mission  are  such  that 
the  classification  of  CCMA  is  not  useful  in 
characterizing  RAM  of  a  system,  then  step 
2  may  be  omitted  from  the  FD/SC. 

♦  During  a  test  incident,  there  will  often  be 
test-peculiar  time,  i.e.,  time  taken  by  the  test 
administrators,  as  distinguished  from  the 
test  players,  for  analysis  and  diagnoses.  This 
test-peculiar  time  should  be  excluded  from 
the  maintenance  times. 

♦  The  crew  maintenance  times  are  usually 
excluded  from  maintainability  parameters; 
however,  this  should  be  reviewed  to 
determine  applicability. 

Operational  Mission  Failure 

The  third  step  in  the  classification  process 
is  to  determine  if  the  incident  was  an 
OMF.  If  the  incident  was  a  malfunction 
that  caused,  or  could  have  caused,  the 
inability  to  perform  one  or  more  mefs,  it 
should  be  scored  as  an  OMF.  In  addition, 
if  the  incident  is  a  critical  or  catastrophic 
hazard  to  personnel  or  equipment,  it 
should  be  scored  as  an  OMF. 

♦  If  maintenance  is  needed  to  restore  the  loss 
of  a  mef,  then  the  OMF  will  also  be  scored 
as  an  EMA  and  an  UMA. 

♦  If  the  malfunction  is  caused  by  another, 
simultaneous  malfunction,  the  latter  will 
be  scored  an  OMF  and  the  former  will  be 
regarded  as  a  secondary  failure  and  will  not 
be  scored. 

♦  If  the  malfunction  is  such  that  the  repair 
can  be  deferred  and  the  mission  can  (safely) 
be  continued,  the  incident  is  not  scored  an 
OMF.  It  will  be  scored  on  its  own  merits 
under  succeeding  steps. 

♦  If  the  system  has  two  components  or 
assemblies,  one  of  which  is  redundant 
to  the  other  at  all  times,  an  OMF  is  not 
scored  unless  both  are  down  at  the  same 
time.  However,  if  the  redundancy  is  not  full 
time,  a  failure  of  the  primary  component 


is  generally  scored  an  OMF  regardless  of 
the  status  of  the  backup  item  at  the  time 
of  the  incident.  Exceptions  to  this  rule 
can  be  made  on  a  case-by-case  basis  if  the 
redundancy  is  nearly  full-time. 

♦  The  recurrence  of  CCMAs  within  a 
limited  period  of  time  may  warrant  the 
classification  of  a  group  of  incidents  as  an 
OMF.  For  example,  “The  recurrence  of  two 
or  more  CCMAs  within  an  hour,  or  four 
(or  more)  with  an  8-hour  mission  will  be 
classified  as  an  OMF.” 

♦  Critical  or  catastrophic  hazards  are  defined 
in  MIL-STD-882  series,  System  Safety 
Program  Requirements. 

Essential  Maintenance  Action 

All  EMAs  are  also  classified  as  UMAs. 

For  some  systems  that  lack  redundant 
features  and  for  which  the  performance 
is  not  affected  by  “special  conditions”  the 
classification  of  EMA  can  be  omitted. 

Unscheduled  Maintenance  Actions 

Any  incident  classified  in  steps  2-4,  or 
any  maintenance  that  does  not  qualify  as 
a  Scheduled  Maintenance  Action  (SMA). 
In  other,  words  any  maintenance  that  does 
not  qualify  as  an  SMA  the  maintenance 
must  be  prescribed  by  an  equipment 
publication;  and,  there  must  be  enough 
latitude  in  the  time  for  the  performance  of 
the  maintenance  that  it  can  be  done  in  a 
slack  period  between  missions. 

Incident  Changeability 

The  following  is  a  description  of  each 
chargeability  category. 

♦  Hardware.  This  category  includes  not 
only  malperforming  hardware  but  also 
personnel-related  incidents  that  are 
attributable  to  the  hardware’s  design.  For 
example,  if  the  device  has  an  exposed  on/ 
off  toggle  switch  that  is  easily  tripped 
inadvertently,  an  unintended  power  down 
of  the  equipment  may  be  charged  to 

the  hardware  vice  the  crew.  Hardware 
chargeability  may  be  further  broken  down 
into  Government-furnished  hardware  and 
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contractor  furnished  hardware. 

♦  Software.  This  category  applies  to  contractor 
and  Government-furnished  software 

that  malfunctions.  Similar  to  hardware, 
personnel-related  incidents  that  are 
attributable  to  the  software  design  may  be 
charged  to  the  software  vice  the  crew. 

♦  Care  should  be  taken  to  distinguish  between 
genuine  software  reliability  problems 

and  simply  improperly  designed  software 
incapable  at  any  time  of  executing  a  given 
task. 

♦  Care  should  also  be  taken  in  defining  what 
software  is  part  of  the  system  under  test 
and  what  software  is  peripheral  events 
(associated).  Application  software  is  usually 
treated  as  “support  equipment.” 

♦  Crew 

♦  Maintenance  Personnel 

♦  Manuals.  These  are  incidents  that  are 
attributable  to  misleading,  incorrect,  or 
nonexistent,  but  needed,  information. 

♦  Training.  These  are  incidents  that  are 
attributable  to  misleading,  incorrect,  or 
nonexistent,  but  needed,  information. 

♦  Support  Equipment.  These  are  incidents 
caused  by  special  and  common  tools  and 
Test  Measurement  Diagnostic  Equipment, 
spares,  repair  parts,  the  associated  software, 
and  sometimes  power  sources. 

♦  Accidents.  This  category  includes  only  those 
accidents  that  are  not  occasioned  by  the 
design  of  the  system.  Incidents  that  should 
not  be  accounted  for  as  accidents  are  those 
due  to  inadequate  training,  inadequate 
warning  in  the  manual,  and/or  careless 
operation.  These  would  be  captured  under 
manuals,  training,  or  crew. 

♦  Unknown.  These  are  incidents  that  cannot 
be  charged  to  one  of  the  above  categories. 
This  category  is  sometimes  helpful  in 

the  characterization  of  communications 
networks  in  which  there  are  “spontaneous 
remissions”  of  a  malfunction.  The  unknown 
category  has  the  potential  for  misuse, 
therefore  it  should  be  used  as  a  last  resort  in 
chargeability  (TRADOC/AMC  1987). 


Hazard  Severity  Assessment 

The  hazard  severity  categories  are  as 

follows: 

♦  Catastrophic  (I):  Could  result  in  death, 
permanent  total  disability,  loss  exceeding 
$1M,  or  irreversible  severe  environmental 
damage  that  violates  law  or  regulation. 

♦  Critical  (II):  Could  result  in  permanent 
partial  disability,  injuries,  or  occupational 
illness  that  may  result  in  hospitalization 
of  at  least  three  personnel,  loss  exceeding 
I200K  but  less  than  $1M,  or  reversible 
environmental  damage  causing  a  violation 
of  law  or  regulation. 

♦  Marginal  (III):  Could  result  in  injury  or 
occupational  illness  resulting  in  one  or  more 
lost  work  days,  loss  exceeding  I10K  but  less 
than  I200K,  or  mitigatible  environmental 
damage  without  violation  of  law  or 
regulation  where  restoration  activities  can 
be  accomplished. 

♦  Negligible  (IV):  Could  result  in  injury 
or  illness  not  resulting  in  a  lost  work  day, 
loss  exceeding  I2K  but  less  than  $10K, 
or  minimal  environmental  damage  not 
violating  law  or  regulation  (DOD  2000). 

Hazard  Probability  Assessment 

The  hazard  probability  categories  are  as 

follows: 

♦  Frequent  (A):  Likely  to  occur  often  in 
the  life  of  an  item,  with  a  probability  of 
occurrence  greater  than  0.1  in  that  life. 

♦  Probable  (B):  Will  occur  several  times  in 
the  life  of  an  item,  with  a  probability  of 
occurrence  less  than  0.1  but  greater  than 
0.01  in  that  life. 

♦  Occasional  (C):  Likely  to  occur  some  time 
in  the  life  of  an  item,  with  a  probability  of 
occurrence  less  than  0.01  but  greater  than 
0.001  in  that  life. 

♦  Remote  (D):  Unlikely  but  possible  to  occur 
in  the  life  of  an  item,  with  a  probability  of 
occurrence  less  than  0.001  but  greater  than 
0.000001  in  that  life. 

♦  Improbable  (E):  So  unlikely,  it  can  be 
assumed  occurrence  may  not  be  experienced, 
with  a  probability  of  occurrence  less  than 
0.000001  in  that  life  (DOD  2000). 
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Test  Incident  Scoring 
Flowchart 
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6-3.  Modeling  and  Simulation  and  the 
Verification ,  Validation ,  and  Accreditation  Process 


This  chapter  provides  an  introduction  to 
the  topic  of  Modeling  and  Simulation 
(M&S),  MCOTEA’s  purpose  in  using 
M&S,  and  the  process  for  its  verification, 


validation,  and  accreditation  ( W&A). 
Table  6-3-1  provides  an  overview  of 
definitions  the  reader  will  need  for  this 
chapter. 


Table  6-3-1 .  Quick-look  Definitions 


Model 

Any  physical,  mathematical,  or  otherwise  logical  representation  of  a  system,  entity,  phenomenon,  or  process. 

Simulation 

Imitates  the  operation  of  a  real-world  process  over  time.  Composed  of  one  or  more  models  and  governed  by 
a  set  of  assumptions  about  system  operation. 

Simulator 

Device  used  to  artificially  duplicate  real  world  conditions.  Typically  used  as  a  training  device. 

Stimulator 

Causes  real-world  response  to  simulated  inputs. 

M&S  Developer 

Individual,  group,  or  organization  that  develops  or  modifies  an  M&S  in  accordance  with  design  requirements 
and  specifications.  Can  also  be  responsible  for  executing  the  Configuration  Management  Plan  and  the  V&V 
Plan. 

M&S  Proponent 

Organization  that  ensures  that  the  M&S  satisfies  the  requirements,  develops  the  V&V  Plan,  performs  V&V, 
develops  reports,  ensures  CM,  and  ensures  sufficient  information  for  M&S  accreditation.  If  MCOTEA  resources 
VV&A,  MCOTEA  is  the  proponent.  For  M&S  MCOTEA  uses  for  OT&E,  the  Program  Office  is  generally  the 
proponent,  supported  by  the  Simulation  Control  Panel  (SCP). 

Simulation  Control  Panel 

Provides  independent  technical  expertise.  Helps  the  Operational  Test  (OT)  Accreditation  Agent  (ACA)  and 
Developmental  Test  (DT)  ACA  understand  model  functionality;  ensures  that  the  M&S  Developer  delivers  the 
intended  M&S  capabilities;  and  assists  in  V&V  activities. 

M&S  User 

Individual,  group,  or  organization  that  uses  the  results  or  products  from  a  specific  application  of  M&S.  For  all 
uses  of  M&S  and  associated  data  in  MCOTEA  test  and  evaluation,  MCOTEA  is  the  M&S  User. 

Subject  Matter  Expert 
(SME) 

An  individual  who,  by  virtue  of  education,  training,  or  experience,  has  expertise  in  a  particular  technical  or 
operational  discipline,  system,  process,  or  M&S. 

Specific  Intended  Uses 
(SIU) 

The  SlUs  are  a  statement  of  how  the  test  team  expects  to  use  M&S  results  in  support  of  the  MCOTEA 
evaluation;  they  represent  M&S  support  essential  to  the  MCOTEA  OT&E. 

Verification 

Process  of  determining  that  a  model  or  simulation  and  their  data  accurately  represent  the  developer's 
description  and  specifications. 

Validation 

Process  of  determining  the  degree  to  which  a  model  or  simulation  and  their  data  accurately  represent  the 
real  world  based  on  the  model's  intended  uses. 

V&V  Agent 

Individual,  group,  or  organization  that  performs  the  verification,  validation,  or  both.  M&S  Proponent 
designates.  If  the  M&S  Developer  functions  as  the  Agent,  an  independent  entity  (such  as  the  SCP)  should 
check  the  work. 

Accreditation 

Official  certification  that  a  model  or  simulation  and  their  data  are  acceptable  for  a  specific  purpose. 

Accreditation  Authority 
(AA) 

Individual  or  organization  responsible  for  approving  use  of  a  model,  simulation,  or  federation  of  such  for  a 
particular  application.  MCOTEA's  AA  is  the  Director  or  a  designee. 

Accreditation  Agent  (ACA) 

Individual,  group,  or  organization  that  conducts  an  accreditation  assessment  for  an  M&S  application.  At 
MCOTEA,  the  COT  designates  the  ACA. 

Accreditation  Criteria 

Accreditation  criteria  are  the  set  of  standards  that  must  be  met  in  order  for  the  M&S  to  be  accredited  for  a 
particular  use. 
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Deciding  to  Use  M&S 

When  conducting  operational  testing  and 
evaluation,  it  is  often  useful  to  consider 
supplementing  the  data  obtained  during 
live  testing  with  data  supported/generated 
by  M&S.  From  MCOTEA’s  perspective, 
M&S  may  offer  an  opportunity  to  examine 
the  system  under  test  in  ways  that  are 
operationally  useful,  but  not  otherwise 
feasible  during  an  operational  test.  For 
example,  firing  an  anti-radiation  missile 
(ARM)  at  a  radar  to  understand  its  ability 
to  detect  a  threat  might  not  be  practical 
during  operational  testing,  but  if  an 
M&S  could  simulate  an  ARM  threat  and 
stimulate  the  radar  satisfactorily,  the  test 
team  could  evaluate  the  radar’s  ability 
to  detect  the  threat  as  well  as  the  radar 
operators’  reaction  to  the  threat. 

MCOTEA  uses  M&S  to  augment,  but  not 
replace,  operational  testing.  M&S  can  be 
used  to 

♦  help  design  tests 

♦  predict  what  happens  during  tests 

♦  provide  stimulation  during  tests 

♦  use  test  data  as  input  to  examine  outcomes 
that  cannot  be  directly  tested 

♦  generate  data  to  supplement  data  generated 
during  tests 

Particular  reasons  to  use  M&S  include 
the  lack  of  test  asset  availability,  lack  of 
sufficient  time  to  generate  adequate  data, 
test  range  limitations,  cost,  and  safety 
considerations. 

Public  law  restricts  the  use  of  M&S 
in  OT&E  such  that  the  results  of  the 
operational  evaluation  cannot  be  “. . . 
based  exclusively  on  computer  modeling; 
simulation;  or  an  analysis  of  system 
requirements,  engineering  proposals, 
design  specifications,  or  other  information 
contained  in  program  documents”  (Title  10 
USC  2399).  In  addition, “M&S  shall  not 
replace  the  need  for  OT&E  and  will  not  be 
the  primary  evaluation  methodology”  (SNI 
5000.2,  section  5. 4.7.9). 


Within  these  constraints,  M&S  can 
provide  a  powerful  way  for  MCOTEA  to 
supplement  the  information  derived  from 
operational  testing. 

The  decision  to  use  M&S  occurs  early 
in  the  OT&E  planning  stages,  before 
the  SEP  is  finalized  and  in  conjunction 
with  the  Program  Office  and  the  T&E 
WIPT’s  development  of  the  Test  and 
Evaluation  Strategy  (TES).The  overall 
T&E  approach  should  define  the  types 
of  data  that  can  be  expected  from  each 
potential  test  venue.  Real  test  data  is 
generally  preferable  to  M&S  data;  however, 
any  data  required  by  the  overall  evaluation 
that  cannot  reasonably  be  gathered  during 
a  test  event  is  a  candidate  for  M&S 
support.  MCOTEA  does  not  generally 
develop  models  or  simulations.  Sources  for 
obtaining  appropriate  M&S  are  listed  later 
in  this  section. 

Expanded  Definitions  of  M&S 

Physical  models  used  in  operational  testing 
include  tank  hulls,  armor  plating,  humans, 
and  buildings.  MCOTEA  once  used  a 
Rocket  Propelled  Grenade  (RPG)  in  an 
operational  test  to  replicate  the  appropriate 
visual  signature  of  an  enemy  role  player. 
This  physical  model  had  all  of  the  visual 
features  of  an  RPG,  but  like  all  models 
it  was  not  without  limitations:  the  RPG 
could  not  be  fired  and  its  weight  was  not 
representative  of  a  real  RPG.  Regardless  of 
these  limitations,  the  physical  model  was 
very  useful  in  the  operational  test  because 
the  limitations  had  no  real  effect  on  how 
the  model  was  used  in  the  test. 

Mathematical  models  represent  aspects  of 
a  system.  For  example,  the  Reliability  of  a 
component  may  be  modeled  using  equation 
R(t)=e_t/MTBF  ,  where  R(t)  is  the  probability 
of  operating  for  a  time  t  without  a  failure 
and  MTBF  is  Mean  Time  Between 
Failures  for  the  component. 


"Models  include 
any  physical, 
mathematical,  or 
otherwise  logical 
representation  of 
a  system,  entity, 
phenomenon, 
or  process. 
Simulations 
include  a  method 
ofimplementing 
a  model  over 
time"  (SNI 
5200.38A). 
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A  simulation  imitates  the  operation  of  a 
real-world  process  or  system  over  time.  The 
simulation  generates  an  artificial  history  for 
a  system.  The  data  composing  the  artificial 
history  is  then  used  to  draw  inferences 
about  the  behavior  and  characteristics  of 
the  real  system.  Simulations  are  composed 
of  one  or  more  models  and  are  governed 
by  a  set  of  assumptions,  inherent  in  the 
models  and  the  way  they  are  applied, 
concerning  the  system’s  operation.  For 
example,  if  the  mathematical  model  in  the 
equation  above  were  used  in  a  simulation, 
the  simulation  would  assume  that  the 
probability  of  system  failures  can  be 
modeled  using  an  exponential  distribution. 
The  models  composing  a  simulation  are 
often  used  through  a  computer  program, 
thus  giving  rise  to  a  computer  model. 

A  simulator  is  a  special  case  of  a 
simulation.  A  simulator  is  a  device  used  to 
artificially  duplicate  real  world  conditions 
so  that  its  operators  can  practice  reacting 
to  those  conditions.  The  simulator 
represents  an  actual  operational  system  to 
varying  degrees  of  fidelity  and  requires  the 
operators  to  interact  with  the  simulator 
much  like  they  would  interact  with  the  real 
system. 

A  stimulator  causes  a  real-world  response 
to  simulated  inputs  in  a  system  or  causes 
a  corresponding  real-world  reaction  by 
an  operator.  Stimulators  are  generally 
associated  with  test  execution  and  are  used 
to  enable  the  examination  of  operationally 
relevant  responses  and  reactions  that 
otherwise  could  not  be  tested.  In  this  role, 
stimulators  are  a  cost-effective  means  to 
test  operational  aspects  of  a  system  the 
testing  of  which  would  otherwise  be  cost 
prohibitive,  unrealistic,  or  hazardous. 

Although  this  section  defines  several 
types  of  models,  MCOTEA  will  generally 
be  concerned  with  computer  models. 
Therefore,  this  chapter  is  written  assuming 
the  M&S  is  a  computer  program/model. 


Using  M&S  in  the  Test  Process 

MCOTEA  uses  M&S  to  minimize  risk 
by  leveraging  test  venues  and  scenarios 
in  a  way  that  maximizes  the  information 
obtained  in  test  planning,  execution,  and 
evaluation. 

During  Test  Planning,  M&S  can 

♦  Help  develop  scenario  and  test  setup 

♦  Predict  outcomes  before  testing  occurs 
During  Test  Execution,  M&S  can 

♦  Stress  systems  under  test  with  large 
numbers  and  higher  densities  than  feasible 
during  actual  testing 

♦  Present  test  situations  that  could  not  safely 
or  practically  be  done  during  actual  testing 

♦  Present  enemy  threats,  systems,  or  counter 
measures  not  otherwise  available  during 
actual  testing 

During  Evaluation,  M&S  can 

♦  Examine  alternative  environments  and 
conditions 

♦  Examine  the  implications  of  system 
deficiencies  and  test  limitations 

♦  Apply  test  data  to  conditions,  subjects,  and 
scenarios  that  cannot  otherwise  be  safely  or 
practically  tested 

Requirement  for  VV&A 

DOD  policy  states  that  all  models, 
simulations,  and  associated  data  used 
to  support  DOD  processes,  products, 
and  decisions  undergo  verification  and 
validation  throughout  their  lifecycles 
(DODI  5000.61).  In  general  MCOTEA 
does  not  perform  V&V ;  MCOTEA’s 
responsibility  is  to  accredit  the  M&S. 

In  order  to  be  used  in  MCOTEA  test 
and  evaluation,  all  M&S  must  undergo 
accreditation.  MCOTEA  may  use  M&S 
in  some  of  the  following  ways: 

♦  test  assets 

♦  test  stimulators 

♦  test  planning  aids 

♦  pre/posttest  analysis  tools 

♦  the  tools’  input  data 

♦  the  tools’  produced  data 
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Once  the  test  team  decides  that  M&S 
support  is  needed,  the  MCOTEA  COT 
appoints  an  Accreditation  Agent  (ACA) 
to  the  team.  The  ACA  is  responsible  for 
organizing  MCOTEA’s  accreditation 
assessment  and  obtaining  all  information 
necessary  to  support  an  accreditation 
decision. 

Developing  Specific  Intended 
Uses  and  Accreditation 
Criteria 

The  first  task  undertaken  by  the  ACA 
in  conjunction  with  the  test  team  is  to 
determine  the  Specific  Intended  Uses 
(SIU),  statements  of  how  the  test  team 
expects  to  use  M&S  results  in  support  of 
the  MCOTEA  evaluation;  they  represent 
M&S  support  essential  to  the  MCOTEA 
OT&E.  After  identifying  the  data  that 
must  come  from  M&S,  the  test  team  can 
define  the  SIUs. 

SIUs  generally  begin  as  high-level 
requirements  that  become  more  detailed  as 
the  test  program  develops.  The  earlier  and 
more  precisely  the  SIUs  can  be  stated,  the 
better.  In  the  end,  if  the  M&S  cannot  be 
accredited  for  certain  SIUs,  the  unachieved 
SIUs  will  generally  represent  limitations  to 
the  OT&E. 

An  effective  SIU  provides  detailed 
information  to  the  M&S  developers.  This 
clarity  in  expectations  helps  the  developers 
deliver  what  is  needed;  they  know  the 
software  development  goal  at  an  early 
stage. 

This  clarity  in  purpose  also  helps  the 
ACA  establish  the  accreditation  criteria, 
used  to  ascertain  whether  the  models  are 
able  to  deliver  the  SIU  with  sufficient 
fidelity.  Accreditation  criteria  are  the  set 
of  standards  that  must  be  met  in  order  for 
the  M&S  to  be  accredited  for  a  particular 
use.  The  criteria  should  include  quantitative 
standards  to  the  maximum  extent  possible 
and  should  be  revisited  periodically  to 
ensure  that  they  remain  appropriate  and 


sufficient  for  the  application.  (Typically, 
the  acceptable  Verification  of  the  M&S  is 
also  an  accreditation  criterion,  but  it  is  not 
associated  with  an  SIU.) 

Locating  the  Right  M&S 

Once  the  required  SIUs  are  determined, 
the  MCOTEA  test  team  in  conjunction 
with  the  Program  Office  select  a  source  for 
the  M&S  support.  When  M&S  support 
is  required  for  both  developmental  and 
operational  testing,  it  might  be  reasonable 
to  use  the  same  M&S  to  support  both.  In 
any  case,  the  required  M&S  support  can  be 
obtained  either  by  using  existing  models  or 
generating  new  ones. 

Existing  Models 

The  following  sources  are  useful  when  an 
existing  or  generic  model  can  satisfy  an 

SIU: 

♦  DOD  M&S  Catalog  (https://MSCatalog. 
osd.mil),  based  on  DODI  5000.61. 

♦  SURVIAC — the  DOD  institution 
responsible  for  collecting,  analyzing,  and 
disseminating  information  related  to 

all  aspects  of  survivability  and  lethality 
for  aircraft,  ground  vehicles,  ships,  and 
spacecraft.  SURVIAC  maintains  several 
approved  models.www.bahdayton.com/ 
surviac/ 

♦  Human  Effects  Center  of  Excellence 
(HECOE) —  maintains  several  models 
designed  to  address  the  effects  of  certain 
stimuli  on  humans  under  various  conditions. 
HECOE  is  located  at  Brooks  City-Base, 
San  Antonio,  TX. 

The  test  team  must  research  their  particular 
area  of  interest  when  searching  for  useful 
existing  models.  The  team  must  also 
consider  the  following  points: 

♦  The  accreditation  process  for  an  existing 
model  should  leverage  previous  V&V  efforts 
to  the  greatest  possible  extent,  but  the  level 
of  vigor  with  which  existing  models  have 
been  verified  and  validated  varies  widely. 

♦  An  existing  model  might  save  development 
time  and  money,  but  MCOTEA  must 
still  accredit  it  for  SIUs  using  appropriate 
accreditation  criteria. 


An  ineffective  SIU 
might  read:"The 
M&S  will  be  used 
to  stimulate  the 
radar  by  producing 
performance  data 
under  a  wide  range 
of  scenarios." 

An  effective  SIU 
might  read: "The 
M&S  will  be  used  to 
stimulate  the  radar 
by  modeling  the 
radar  cross  section, 
delivery  profile, 
and  kinematic 
performance  of  a 
threat  anti-radiation 
missile  and  overlay 
this  data  onto 
the  data  stream 
containing  the 
returns  of  the  actual 
radar  during  the  live 
play  of  operational 
testing." 
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TheVV&A  process  is 
rooted  in  confidence: 
"Confidence  in  a 
particular  model 
or  simulation  must 
be  justified  before 
its  results  are  used 
to  make  decisions 
involving  large  sums 
ofmoneyorriskto 
human  life"  (SNI 
5200.40). 


All  models  using 
stochastic  processes 
should  also  provide 
guidance  on  the 
number  of  iterations 
necessary  considering 
runtime,  confidence 
levels,  and  output 
stability. 


New/ Modified  Models 

Creating  new  M&S  may  best  be  done  by  the 
system  developer  if  the  M&S  will  be  used  to 
predict  system  performance  or  to  stimulate  the 
system,  and  if  the  system  developer  is  capable 
of  creating  the  M&S.  The  system  developer 
knows  the  system  capabilities  and  is  probably 
best  positioned  to  convey  this  knowledge  to  an 
M&S.  However,  the  test  team  must  be  aware 
that  the  system  developer  will  be  motivated 
to  make  their  M&S  as  well  as  their  system 
look  good.  Therefore,  the  test  team  must  be 
vigilant  in  monitoring  the  M&S’s  development, 
verification,  validation,  and  quality  assurance. 

The  test  team  should  revisit  the  Simulation 
Interface  Units  (SIU)  after  deciding  on  a 
model  because 

♦  increased  familiarity  with  test  concepts  may 
inspire  the  need  for  different/additional 
SIUs 

♦  the  test  team  might  want  to  take  advantage 
of  the  supplemental  capabilities  of  a 
previously  unidentified  existing  model 

M&S  Funding  and  Schedule 
Requirements 

Once  the  test  team  understands  the 
required  level  of  M&S  support  for 
OT&E,  the  OTPO  notifies  the  Program 
Office  of  MCOTEA’s  requirements 
and  plans  so  that  sufficient  funding  for 
model  development  and  V&V  efforts 
is  allotted.  This  funded  support  must 
include  the  technical  expertise  required  to 
develop,  manage,  operate,  verify,  validate, 
accredit,  and  apply  the  results  of  the  M&S 
application.  In  addition,  the  funded  support 
should  include  the  Simulation  Control 
Panel  (explained  later  in  this  chapter) 
and  any  supplemental,  independent  SME 
support  required  to  ensure  that  final 
W&A  requirements  are  met.  The  OTPO 
and  ACA  must  assess  the  adequacy  and 
technical  soundness  of  the  PO’s  approach 
to  satisfying  MCOTEA  M&S  and  W&A 
requirements. 

The  program’s  schedule  must  include  a 
realistic  amount  of  time  for  the  required 


level  of  W&A,  at  a  minimum  several 
months.  The  OTPO  and  the  MCOTEA 
Test  Manager  must  be  kept  apprised  of  the 
M&S  schedule  for  overall  test  planning 
purposes.  If  the  M&S  will  be  new,  realistic 
time  must  also  be  provided  for  model 
development.  Constraints  on  schedule 
and  cost  exist  for  any  program,  but  the 
MCOTEA  SIUs  represent  data  essential 
for  overall  system  evaluation.  To  the  extent 
that  funding  or  schedule  are  not  adequately 
provided,  any  neglected  SIUs  become 
limitations  to  the  OT&E. 

Verifying,  Validating,  and 
Accrediting  M&S 

Although  MCOTEA’s  role  in  the  W&A 
process  is  to  provide  the  accreditation, 
generally  not  the  V&V,  being  familiar 
with  the  entire  W&A  process  allows 
MCOTEA  to  understand  the  M&S’s 
capabilities,  reduce  the  risk  associated 
with  using  the  M&S,  and  make  informed 
decisions  about  using  an  M&S  in  support 
of  OT&E. 

Furthermore,  MCOTEA  is  closely 
involved  in  helping  to  determine  V&V 
activities  and  in  witnessing  the  results, 
reinforcing  the  idea  that  understanding 
the  complete  process  is  essential  for 
appropriate  MCOTEA  participation. 

This  section  focuses  on  MCOTEA’s 
accreditation  responsibilities,  supported 
by  essential  information  about  the  V&V 
process.  Annexes  A  and  B  contain  detailed 
information  about  V&V. 

♦  MCOTEA  must  be  confident  that  a 
particular  M&S  is  accurate  and  suitable  for 
the  SIUs 

♦  This  confidence  must  be  based  on  an 
unbiased  assessment  of  the  M&S 

♦  The  justification  of  this  confidence  must 
be  communicated  for  future  use  using 
established  reporting  mechanisms  as 
described  in  this  chapter  (also  see  SNI 
5200.40). 

The  term  VV&A  is  often  used  in  the  context 
of  a  single  activity,  but  the  VV&A  process 


6-38 


Ancillary  Topics 


contains  three  distinct  and  separable  activities 
(DODI  5000.61): 

♦  Verification:  The  process  of  determining 
that  a  model  or  simulation  implementation 
and  its  associated  data  accurately  represent 
the  developer’s  conceptual  description  and 
specifications. 

♦  Validation:  The  process  of  determining 
the  degree  to  which  a  model  or  simulation 
and  its  associated  data  are  an  accurate 
representation  of  the  real  world  from  the 
perspective  of  the  intended  uses  of  the 
model. 

♦  Accreditation:  The  official  certification  that 
a  model  or  simulation  and  its  associated 
data  are  acceptable  for  use  for  a  specific 
purpose. 

General  Uses  of  M&S 
Requiring  Accreditation 

MCOTEA  must  accredit  an  M&S  when  the 
M&S  is  used  to  support  operational  testing 
or  if  the  data  will  be  used  in  the  evaluation  of 
OE,  OS,  OSur.  For  M&S  used  in  DT  (i.e., 
MCOTEA  involvement  is  only  to  determine  if 
the  threshold  has  been  met),  then  the  TEMP 
will  identify  the  agencies  responsible  for 
accreditation. 

MCOTEA  accreditation  applies  to  all 
categories  of  models  and  simulations: 

♦  live,  virtual,  and  constructive  simulations 

♦  unitary,  federated,  or  distributed  simulations 

♦  COTS/GOTS/NDI  software  or  hardware, 
emulators,  and  prototypes 

♦  simulators 

♦  stimulators 
as  well  as 

♦  the  data  needed  to  verify  M&S 
requirements,  validate  the  M&S,  perform 
experiments  on/with  the  M&S,  or  run 
combat  M&S  decision  aids. 

Ideally,  verification  is  an  integral  part  of 
model  development,  meaning  that  the 
model  verification  techniques  are  identified 
a  priori  and  execution  of  these  techniques  is 
documented  as  they  are  performed  during 
model  development.  However,  if  verification 
of  legacy  M&S  was  not  accomplished, 


was  inadequate,  or  was  not  documented, 
MCOTEA  may  need  to  require  additional 
verification. 

Note:  Many  organizations  (including  for- 
profit  contactors,  not-for-profit  contractors, 
universities,  and  various  government 
organizations  and  labs)  develop  models  that 
might  supplement  a  MCOTEA  OT&E.The 
concepts,  techniques,  procedures,  and  standards 
MCOTEA  uses  to  accredit  an  M&S  apply 
to  all  M&S  developers,  regardless  of  their 
pedigree. 

W&A  Stakeholders  and  Their  Roles 

Overall  VV&A  involves  the  following 
stakeholders: 

♦  M&S  User 

♦  Accreditation  Authority 

♦  Accreditation  Agent 

♦  M&S  Proponent 

♦  V&V  Agent 

♦  M&S  Developer 

♦  SME 

The  MCOTEA  stakeholders  in  the  fist  are 
the  M&S  User  (for  M&S  used  in  OT&E), 
the  Accreditation  Authority  (AA),  and  the 
ACA.  It  is  possible  for  MCOTEA  to  be 
the  M&S  Proponent  as  well  if  MCOTEA 
funds  the  M&S,  but  funding  typically 
occurs  through  the  Program  Office. 

Accreditation  Authority 

Generally  speaking,  the  AA  is  the 
organization  or  individual  responsible  for 
approving  the  use  of  a  model,  simulation, 
or  federation  of  models  and  simulations  for 
a  particular  application.  The  AA  ensures 
that  resources  are  available  for  the  W&A 
effort.  For  all  uses  of  M&S  and  associated 
data  in  MCOTEA  tests  or  evaluations, 
the  AA  is  the  Director,  MCOTEA  or  a 
designated  representative.  The  AA’s  chief 
responsibilities  are  as  follows: 

♦  Determine  the  appropriateness  of  an  M&S 
for  the  required  SIUs 

♦  Ensure  that  adequate  V&V  has  occurred 


Software  applications 
requiring  MCOTEA 
accreditation  do  not 
include  the  software 
that  is  an  integral 
part  of  the  system 
under  test,  including 
firmware  and  other 
software  required 
to  drive  the  system. 
MCOTEA  expects  this 
type  of  soft  ware  to  be 
verified  along  with  the 
verification  of  other 
system  specifications 
during  developmental 
testing. 
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before  using  the  M&S  in  OT&E 

♦  Require  Independent  Verification  and 
Validation  of  the  M&S  if  needed 

♦  Sign  the  Accreditation  Decision  Letter 
based  on  the  Accreditation  Report 
(documentation  is  explained  in  detail  later 
in  this  chapter) 

Accreditation  Agent 

Designated  by  the  MCOTEA  COT,  the 
fundamental  job  of  the  ACA  is  to  gather 
the  information  the  AA  needs  to  make 
an  informed  accreditation  decision.  The 
ACA  fulfills  an  essential  role  and  should 
function  as  an  adjunct  to  the  test  team, 
without  other  team  responsibilities.  (If 
M&S  will  be  used  to  support  DT,  the 
PMO  will  assign  its  own  Accreditation 
Agent  to  the  program.)  The  COT  can 
designate  an  independent  organization 
or  assign  a  staff  member  or  support 
contractor  as  MCOTEA’s  Accreditation 
Agent;  ideally,  the  ACA  will  have 
experience  in  V&V  to  help  provide  an 
understanding  of  the  entire  W&A 
process.  The  ACA’s  chief  responsibilities 
are  as  follows: 

♦  Ensure,  early  in  the  W&A  process,  that 
the  M&S  Developer  and  M&S  Proponent 
are  familiar  with  MCOTEA  V&V 
requirements. 

♦  Represent  the  test  team  at  all  internal  and 
external  discussions  of  M&S  in  support  of 
OT&E. 

♦  Lead  the  test  team  discussion  to  determine 
the  desired  SIUs.  Leading  the  discussion 
gives  the  ACA  a  deep  understanding  of  the 
T&E  strategy,  the  role  each  SIU  is  intended 
to  perform  in  the  system  evaluation,  and  the 
corresponding  importance  of  each  SIU. 

♦  Lead  the  MCOTEA  effort  to  determine 
the  appropriate  M&S  for  OT&E 

♦  Conduct  the  research  to  support  which 
M&S  to  use 

♦  Provide  the  research  results  to  the  OTPO 
for  selecting  the  best  M&S  options 

♦  Function  as  the  resident  MCOTEA 


expert  on  the  M&S  being  considered  for 
accreditation 

♦  Become  familiar  with  M&S  assumptions, 
capabilities,  limitations,  and  history 

♦  Monitor  the  resolution  of  technical  issues 
to  ensure  that  the  M&S  will  be  capable  of 
executing  the  MCOTEA  SIUs 

♦  Participate  in  the  Simulation  Control  Panel 

♦  Prepare  an  Accreditation  Plan  and 
Accreditation  Report 

In  addition,  the  ACA  coordinates  all 
required  MCOTEA  participation  in 
inspections,  analyses,  demonstrations, 
and  tests  in  support  of  M&S  accuracy 
and  requirements  verification;  M&S 
capability  validation;  supporting  data 
V&V ;  and  validation  that  the  M&S 
satisfies  the  accreditation  criteria  of  all 
MCOTEA  SIUs. 

Other  Important  Vi iV  Roles 

The  following  section  defines  other,  non- 
MCOTEA  stakeholders  and  their  roles  in 
the  V&V  process. 

M&S  Developer.  The  individual,  group, 
or  organization  responsible  for  developing 
or  modifying  an  M&S  in  accordance 
with  a  set  of  design  requirements  and 
specifications.  The  M&S  Developer  can 
also  be  responsible  for  executing  the 
Configuration  Management  (CM)  Plan 
and  the  V&V  Plan. 

M&S  Proponent.  The  organization  with 
primary  responsibility  for 

♦  ensuring  that  the  M&S  satisfies  the  stated 
requirements 

♦  developing  the  V&V  Plan  and  Report 

♦  performing  V&V  activities 

♦  ensuring  effective  CM 

♦  ensuring  that  sufficient  information  is 
gathered  to  support  the  M&S  accreditation 

It  is  not  unusual  for  the  M&S  Proponent 
to  contract  with  the  M&S  Developer 
to  generate  drafts  of  the  CM  Plan,  the 
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V&V  Plan,  and  the  V&V  Report  for 
independent  review  and  approval.  The  M&S 
Proponent  is  responsible  for  delivering  this 
documentation  to  the  DOD  M&S  Catalog 
(https:/ /M  SCatalog.  osd.  mil) . 

M&S  User:  The  individual,  group,  or 
organization  that  uses  the  results  or 
products  from  a  specific  application  of  an 
M&S.  For  all  uses  of  M&S  and  associated 
data  in  MCOTEA  tests  or  evaluations, 
MCOTEA  is  the  M&S  User. 

Subject  Matter  Expert :  An  individual 
who,  by  virtue  of  education,  training,  or 
experience,  has  expertise  in  a  particular 
technical  or  operational  discipline,  system, 
process,  or  M&S.  SMEs  are  on  the 
Simulation  Control  Panel  (SCP)  primarily 
to  help  the  MCOTEA  ACA  and  the  DT 
ACA  gain  an  understanding  of  model 
functionality,  ensure  the  M&S  Developer 
delivers  the  intended  M&S  capabilities,  and 
assist  in  verification  and  validation  activities. 

Verification  and/ or  Validation  Agent: 

The  individual,  group,  or  organization 
designated  by  the  M&S  Proponent  to 
perform  the  verification,  validation,  or 
both,  of  a  model,  simulation,  or  federation 
of  models  and  simulations,  and  their 
associated  data.  If  this  is  the  M&S 
Developer,  the  V&V  should  be  checked  by 
an  independent  entity  such  as  the  SCP. 

Further  detailed  information  about 
stakeholders’  individual  roles  and 
responsibilities  can  be  found  in  the  M&S 
W&A  Implementation  Handbook. 

VV&A  Process  in  Total 

The  following  two  flow  charts  present  the 
basic  W&A  (focusing  on  accreditation) 
process,  beginning  with  seven  steps 
generic  to  any  accreditation  once  the  test 
team  decides  to  use  M&S.  Figure  6-3-2 
traces  the  process  of  accrediting  a  new  or 
modified  model.  Figure  6-3-3  depicts  the 
accreditation  process  for  an  existing  model. 


Although  MCOTEA  does  not  generally 
perform  V&V,  the  ACA  supports  the 
determination  of  acceptable  V&V,  and  thus 
must  be  familiar  with  the  V&V  activites 
and  processes. 

The  Accreditation  Process 

The  accreditation  process  applies 
to  a  specific  version  of  an  M&S,  a 
predetermined  set  of  SIUs,  and  everything 
that  supports  the  M&S  to  be  used  in 
OT&E.The  process  can  be  organized  into 
six  parts: 

1.  Assessing  the  M&S  Developer’s 
experience  and  processes 

2.  Assessing  M&S  functionality  and 
assumptions 

3.  Verifying  and  validating  M&S  input  data 

4.  Verifying  requirements  satisfaction  and 
model  accuracy 

5.  Validating  replication  of  the  real  world 

6.  Ensuring  that  accreditation  criteria  for 
each  SIU  are  satisfied 

The  results  of  the  accreditation  effort 
establishes  the  level  of  credibility 
MCOTEA  bestows  on  the  M&S  with 
respect  to  the  SIUs  in  question.  The 
assessment  may  or  may  not  result  in 
MCOTEA  M&S  accreditation  for  all  SIUs. 

Accreditation  Schedule 

In  all  cases,  the  version  of  an  M&S 
intended  to  support  OT&E  must  be  locked 
down  (no  further  changes),  with  SIUs  fully 
accredited,  30  days  before  the  OTRB  (120 
days  before  OT).  If  certain  SIUs  cannot  be 
fully  accredited  by  this  time,  the  test  team 
must  determine  alternative  ways  to  satisfy 
the  requirements  of  the  unaccredited  SIUs 
or  plan  to  report  them  as  Test  Limitations. 
The  test  team  must  brief  alternative  plans 
for  each  unaccredited  SIU  at  the  OTRB. 
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*  Assumes  joint  MCOTEA/PO 
Accreditation  Plan 

Fig.  6-3-2. 

Accrediting  a 
New  or  Modified 
Model 
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*  Assumes  joint  MCOTEA/PO 
Accreditation  Plan 


Fig.  6-3-3. 
Accrediting  an 
Existing  Model 
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Simulation  Control  Panel 

The  primary  purpose  of  the  SCP  is  to 
help  the  OT  ACA  and  the  DT  ACA 
understand  model  assumptions  and 
functionality,  ensure  the  M&S  Developer 
delivers  the  intended  M&S  capabilities, 
and  assist  in  verification  and  validation 
activities. 

If  the  PMO  funds  the  accreditation 
effort,  the  PMO  charters  the  SCP  in 
collaboration  with  the  MCOTEA  ACA. 
In  this  case,  the  SCP  is  composed  of  the 
MCOTEA  ACA,  the  PO  Accreditation 
Agent  representing  DT  SIUs,  the  V&V 
Agent,  PO  SMEs,  the  model  developer, 
and  independent  government  and 
contractor  technical  experts  as  determined 
by  the  PO  and  MCOTEA.  The  PO 
appoints  the  SCP  chair. 

If  MCOTEA  funds  the  accreditation 
effort,  MCOTEA  charters  the  SCP  with 
the  same  membership.  In  principle,  each 
M&S  requires  its  own  SCP.  In  practice, 
if  several  different  but  related  M&Ss 
are  needed  for  a  program,  it  is  often 
convenient  to  include  the  necessary  SMEs 
in  a  single  SCP.  The  SMEs  can  be  domain 
professionals  (such  as  experts  in  what  is 
being  modeled  by  the  M&S,  e.g.,  doctors 
and  system  experts  as  required.) 

The  SCP  meets  periodically  to  review 
and  approve  model  methodologies,  use  of 
algorithms,  model  assumptions,  accuracy 
of  approach,  adequacy  and  applicability  of 
input  data,  model  developer  processes,  and 
documentation.  Based  on  this  activity,  the 
SCP  provides  periodic  reports  of  model 
status,  plans,  and  schedule  to  the  DT 
and  OT  ACAs.  The  chairman  writes  (or 
assigns)  minutes  for  every  meeting  and 
appoints  others  to  write  on  special  topics 
as  needed. 


The  chief  responsibilities  of  the  SCP  are 
as  follows: 

•Serve  as  a  communication  conduit 
between  the  ACAs  and  the  M&S 
developer. 

•Provide  independent  expertise  to  help 
address  important  technical  issues  and 
assist  the  ACAs  in  gathering  relevant 
technical  information. 

•Probe  the  operating  details  of  the  model 
to  understand  model  assumptions  and 
methodologies. 

•Provide  guidelines  for  data  V&V 

§  Why  and  how  the  data  was 
generated  (the  more  detail  the  better) 

§  Any  assumptions  made  in 
generating  the  data. 

•Provide  guidelines  for  the  V&V  Plan 

§  Outline  and  schedule 

§  Any  needed  clarification  of 
accreditation  requirements 

•Review  and  approve  final  V&V  Plan 

•Possibly  require  V&V  techniques  in 
addition  to  those  found  in  the  reference 
(DOD  W&A  RPG,  www.wa.msco.mil, 
Reference  Documents,  V&V  Techniques) 
during  V&V  Plan  execution 

•Provide  guidelines  for  the  V&V  Report 

§  Outline  and  schedule 

§  Any  needed  answers  about  content 
and  distribution 

•Review  and  approve  V&V  Report 

•Deliver  the  V&V  Report  to  DT  and  OT 
ACAs 

Once  the  accreditation  decisions  are  made, 
the  SCP  is  dissolved. 
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Overview  of  the  Accreditation 
Assessment 

A  MCOTEA  accreditation  assessment 
requires  much  more  than  simply 
reading  reports.  It  requires  the  active 
participation  of  the  ACA  as  described 
in  this  section.  It  is  important  that  the 
accreditation  activities  begin  early  in 
a  program  so  the  M&S  Developer  is 
aware  of  MCOTEA  V&V  requirements 
and  can  react  to  them.  The  accreditation 
agent  needs  first-hand  knowledge  of  the 
M&S  V&V  activities  and  techniques, 
the  assumptions  used  by  the  M&S,  and 
an  understanding  of  the  strengths  and 
weakness  of  the  M&S  relative  to  the 
OT&E  SIUs.This  requires  the  early 
and  active  participation  of  the  ACA  in 
V&V  activities.  This  section  describes  the 
types  of  activities  the  MCOTEA  ACA 
is  expected  to  accomplish  to  perform  an 
accreditation  assessment. 

1.  Assessing  the  MElS  Developer’s 
Experience  and  Processes 

Confidence  lies  at  the  root  of  all  actions 
associated  with  verifying,  validating,  and 
accrediting  an  M&S. 

To  generate  the  required  confidence,  the 
ACA  must  first  gather  general  information 


about  the  M&S  Developer’s  software 
development  history  and  processes.  This 
information  will  indicate  the  degree 
to  which  the  M&S  Developer  follows 
established  software  development  and 
software  quality  assurance  procedures, 
leading  to  ACA  awareness  of  how  closely 
the  M&S  Developer’s  work  must  be 
scrutinized.  The  following  issues  are 
examples  of  general  information  that  can 
indicate  the  M&S  Developer’s  competence. 
These  issues  are  not  all  inclusive,  and  the 
OT&E  team/ACA  are  encouraged  to  ask 
additional  questions  that  may  apply  more 
closely  to  their  specific  circumstances. 
Positive  answers  to  all  of  the  following 
issues  will  create  confidence  in  the 
developer’s  ability  to  construct  a  quality 
M&S.  If  the  developer  cannot  satisfactorily 
address  one  or  more  of  the  following  issues, 
additional  scrutiny  by  MCOTEA  may  be 
warranted. 

Historical  error  detection  efficiency.  If  the 
M&S  Developer  has  this  information, 
it  is  probably  valid  for  some  amount  of 
time  after  initial  release.  Error  detection 
efficiency  can  be  calculated  as  (number  of 
errors  detected  before  release  of  software)/ 
(number  of  errors  detected  before  release 
+  number  of  errors  detected  after  release). 
The  higher  the  result  the  better.  The  error 


Relative  Effort  Required  for 
Accreditation  Tasks 


& 

A  £ 

J'J’ 


q? 


NT 

*v  ^ 


Figure  6-3-1.  The  major  categories  of  accreditation  tasks  where  the  horizontal  width  of  each  layer  represents  the  relative 
amount  of  effort  required  to  accomplish  the  given  task.  This  figure  indicates  that  the  "Assessment  of  M&S  functionality 
and  assumptions"  can  be  expected  to  take  roughly  as  much  effort  as  "M&S  validation  activities."  Both  of  these  tasks  can 
be  expected  to  take  roughly  four  times  the  effort  of  the  "Assessment  of  M&S  Developer's  Experience  and  Processes"  tasks. 
This  figure  is  intended  to  be  used  as  a  tool  for  planning  the  accreditation. 


6-45 


Chapter  6 


detection  efficiency  for  some  organizations 
exceeds  99  percent.  In  general,  a  rate 
below  90  percent  indicates  that  the  M&S 
warrants  additional  scrutiny  for  errors. 

Programming  languages  used  in  developing 
the  M&S  and  number  of  software  lines  of 
code  (SLOC)  required  during  programming. 
Estimate  SLOC  for  codes  under 
construction.  The  SLOC  can  be  used  to 
indicate  the  level  of  the  code’s  complexity. 

A  model  comprising  10,000  lines  of  code 
can  be  expected  to  harbor  more  errors  than 
a  model  comprising  hundreds  of  lines. 

Software  development  and  software  quality 
assurance  processes  and  best  practices,  including 
supporting  documentation.  Generally, 
software  quality  is  emphasized  in  an 
organization  when  actual  documented 
processes  exist  and  the  developer  is 
conversant  in  these  processes.  Even  more 
confidence-inspiring  is  an  industry- 
recognized  rating  (such  as  a  mid-  to  high- 
level  CMMI  rating)  of  the  developer’s 
processes.  Lack  of  documented  processes 
or  lack  of  process  awareness  should  be 
considered  a  red  flag. 

Defined  cutoffs  for  code  modifications.  This 
refers  to  modifications  in  the  code’s 
capability,  not  the  correction  of  errors, 
and  has  implications  for  configuration 
management.  Having  clearly  defined 
cutoffs  indicates  awareness  of  the  basic 
tenets  of  configuration  management. 
Vagueness  in  this  area  can  indicate  version 
creep  and  schedule  slippage. 

Verification  process  execution.  The  process 
should  specify  exactly  what  will  be  done 
(e.g.,  module  testing)  and  who  will  do  it 
(e.g.,  an  independent  team  of  software 
engineers).  See  Annex  A  for  verification 
techniques. 

M&S  error  inspection.  Ideally,  outside 
experts  in  the  language  used  to  write  the 
software  should  inspect  each  module  from 
an  independent  perspective. 

SME  availability  to  answer  software 
engineers’  questions.  The  engineers  writing 


the  code  will  inevitably  need  to  ask 
operational  or  technical  questions.  Having 
SMEs  available  will  help  to  minimize 
erroneous  operational  assumptions. 

The  ACA  shall  assess  the  responses  to  all 
these  issues  and  make  an  overall  statement 
pertaining  to  the  quality  of  work  that  can 
be  expected  from  the  M&S  Developer  in 
the  Accreditation  Report. 

2.  Assessing  MEtS  Functionality  and 
Assumptions 

The  ACA  is  not  required  to  know  the 
language  used  in  programming  the  M&S 
under  consideration;  however,  the  ACA 
is  required  to  have  a  good  understanding 
of  what  the  model  is  intended  to  do,  the 
methodology  it  uses,  and  the  assumptions 
made  in  coding  and  running  the  model. 
This  knowledge  can  be  obtained  by  reading 
the  model  description,  user’s  manual, 
Software  Requirements  Specification, 
independent  model  reviews,  and  any  other 
documentation  about  the  model  that  is 
available  and  relevant. 

The  SMEs  on  the  SCP  can  be  very  helpful 
in  understanding  the  right  questions  to  ask 
as  well  as  in  interpreting  the  explanations 
associated  with  those  questions.  Basic 
understanding  of  the  M&S  pays  dividends 
when  the  ACA  witnesses  V&V  events  and 
can  judge  the  significance  of  the  results. 

The  ACA  shall  summarize  the  documents 
reviewed  and  other  steps  taken  to  gain 
an  understanding  of  M&S  functionality 
in  the  Accreditation  Report.  In  addition, 
the  ACA  shall  list  the  major  assumptions 
that  are  made  by  the  M&S  and  state  the 
effect,  if  any,  of  each  assumption  on  the 
performance  of  each  OT&E  SIU  in  the 
Accreditation  Report. 

3.  Verifying  and  Validating  MElS 
Input  Data 

If  the  M&S  uses  any  form  of  input  data 
or  has  parameter  values  hardwired  into  the 
code,  the  ACA  addresses  how  that  data  was 
verified  and  validated.  Even  if  the  M&S 
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is  functioning  perfectly,  accurate  results 
from  the  model  cannot  be  guaranteed  if  the 
data  required  by  the  M&S  has  not  been 
acceptably  V&V’d,  and  the  M&S  has  not 
been  accredited  for  uses  that  depend  on 
non-V&V'd  data.  The  ACA  is  expected  to 
certify  that  the  data  used  by  the  M&S  is 
accurate,  consistent,  and  suitable  for  use 
by  the  M&S.  The  data  must  be  V&V’d  in 
accordance  with  the  “Data  Verification  and 
Validation”  section  of  this  chapter. 

The  specific  activities  addressing  data  V&V 
should  be  documented  in  the  Accreditation 
Report.  Key  information  on  MCOTEA 
expectations  for  data  V&V  and  ACA 
responsibilities  in  this  regard  can  be  found 
in  Annex  B. 

4.  Verifying  Requirements 
Satisfaction  and  Model  Accuracy 

Verification  is  an  assessment  of  how 
well  the  M&S  satisfies  its  software 
requirements  and  how  accurately  the 
M&S  performs.  The  ACA  or  a  suitable 
government  representative  witnesses  M&S 
verification  activity  in  accordance  with 
this  chapter.  The  ACA  provides  an  overall 
assessment  of  the  verification  techniques 
used  and  whether  the  verification  activities 
were  comprehensive  and  thorough  in 
locating  software  errors.  The  ACA  also 
assesses  the  verification  of  software 
requirements  and  comments  on  any  that 
were  not  adequately  verified. 

Verification  efforts  are  expected  to  locate 
errors  in  the  M&S.  Therefore,  the  model  is 
expected  to  undergo  a  great  deal  of  change 
during  verification.  Changes  to  the  M&S 
can  also  be  expected  in  validation.  Once 
the  version  of  the  M&S  that  will  support 
OT&E  is  finalized,  previous  verification 
tests  should  be  rerun  as  a  best  practice. 

The  ACA  shall  summarize  the  verification 
activities  in  the  Accreditation  Report.  Key 
information  on  MCOTEA  expectations 
of  verification  and  ACA  responsibilities  in 
this  regard  can  be  found  in  Annex  A. 


5.  Validating  MElS  Results 

It  is  expected  that  different  and 
complementary  validation  techniques 
will  be  performed  on  the  M&S  to  build 
confidence  that  the  M&S  can  realistically 
and  accurately  support  the  SIUs  needed  to 
support  OT&E.  The  ACA  assesses  each 
validation  technique  used  in  the  overall 
assessment  of  the  M&S  validation.  As 
with  verification,  if  the  M&S  is  changed 
as  a  result  of  a  validation  test,  the  ACA 
describes  the  type  and  adequacy  of  the 
regression  testing. 

The  ACA  summarizes  all  validation 
activity  in  the  Accrediation  Report.  Key 
information  on  MCOTEA  expectations 
for  validation  and  ACA  responsibilities  in 
this  regard  can  be  found  in  Annex  A. 

6.  Ensuring  that  Accreditation 
Criteria  for  Each  SIU  are  Satisfied 

The  ACA  must  assess  the  satisfaction  of 
accreditation  criteria  associated  with  each 
SIU.  The  ACA  assesses  the  adequacy  and 
accuracy  of  the  data  collected  independent 
of  the  M&S  to  support  a  comparison  with 
M&S  results.  The  ACA  then  examines 
whether  the  accuracy  levels  and  confidence 
levels  (if  stated)  in  the  Accreditation 
Criteria  are  met. 

Overall  Assessment 

MCOTEA  accredits  a  specific  version 
of  an  M&S  by  individual  SIU.  Each  SIU 
receives  its  own  accreditation  assessment 
and  recommendation,  based  on  the  results 
of  the  preceding  steps.  However,  going 
directly  to  step  6,  if  the  M&S  fails  to  meet 
an  accreditation  criterion  for  any  SIU,  the 
ACA  cannot  recommend  accreditation  for 
that  SIU.  Conversely,  meeting  accreditation 
criteria  does  not  guarantee  accreditation  for 
that  SIU.  For  example,  the  data  required 
for  input  and  used  to  support  the  SIU  may 
not  be  sufficiently  V&V’d,  or  the  M&S 
may  make  an  inappropriate  assumption 
regarding  the  SIU.  For  each  SIU  that 
passes  its  accreditation  criteria,  the  ACA 
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Conceptual  Model 

According  to  SNI  5200.40,  enclosure  1, 
section  2.b(2),  “The  conceptual  model 
serves  as  a  bridge  between  the  defined 
requirements  and  the  M&S  design, 
providing  the  developer’s  interpretation  of 
the  requirements  to  which  the  model  or 
simulation  will  be  built. ’’The  documented 
conceptual  model  is  constructed  by  the 
M&S  Developer  before  coding  begins 
and  should  contain  the  fundamental 
assumptions  used  by  the  M&S,  the 
availability  of  data  required  by  the  M&S, 
descriptions  of  the  functional  modules 
(e.g.,  subroutines,  objects,  etc.)  in  use 
by  the  M&S,  as  well  as  the  architecture 
used  to  relate  the  functional  modules 
to  one  another  and  to  other  models 
or  simulations.  The  conceptual  model 
describes  what  the  M&S  is  expected  to  do 
along  with  any  supplemental  information 
and  data  and  their  sources.  Although 
MCOTEA  need  not  participate  in  the 
development  of  the  conceptual  model,  the 
information  it  contains  is  important  to  the 
overall  understanding  of  the  M&S.  If  this 
information  is  not  explicitly  contained  in 


something  called  the  conceptual  model, 
it  will  have  to  be  obtained  elsewhere. 
Candidate  alternative  sources  of  the  type 
of  information  typically  found  in  the 
conceptual  model  are  the  User’s  Manual, 
M&S  Description,  M&S  development 
documentation,  and  any  previous  W&A 
documentation. 

The  ACA  should  review  the  Conceptual 
Model  for  completeness  and  to  learn 
about  the  M&S.  The  Conceptual 
Model  is  validated  and  the  validation 
documentation  should  be  available  for 
review.  MCOTEA  need  not  be  part 
of  the  Conceptual  Model  validation, 
but  the  validation  report  should  be 
reviewed  to  ensure  the  M&S  Developer’s 
interpretation  of  the  M&S  requirements 
is  consistent  with  MCOTEA’s. 

If  no  conceptual  model  exists  for  a  legacy 
simulation,  it  should  be  constructed  and 
validated  if  the  simulation  is  modified.  If 
the  conceptual  model  is  constructed  for  a 
modified  M&S,  it  should  cover  both  the 
legacy  portions  and  the  modified  portions 
of  the  M&S. 


considers  all  of  the  preceding  accreditation 
steps  before  recommending  accreditation 
for  that  SIU. 

VV&A  Documentation 
Process 

Table  1  illustrates  the  core  documentation 
produced  for  W&A.  MCOTEA 
is  responsible  for  producing  the 
Accreditation  Plan  and  the  Accreditation 
Report.  The  V&V  Agent  produces  the 
V&V  Plan  and  V&V  Report.  The  normal 
set  of  MCOTEA  templates  for  plans  and 
reports  is  not  used  for  VV&A  because 
the  DOD-established  W&A  process 
calls  for  sharing  information  among  those 
who  verify,  validate,  and  accredit.  A  set 
of  MIL-STD  templates  (MIL-STD 
3022)  is  available  for  this  purpose,  which 
MCOTEA  uses  for  consistency  with  other 
W&A  stakeholders.  Table  1  illustrates  the 


core  documents  that  support  the  W&A 
Process. 

The  Accreditation  Plan  and  the  V&V 
Plan  are  analogous  to  a  TEMP  in 
that  they  set  forth  the  expectations  of 
the  entire  VV&A  process.  The  V&V 
Report  and  the  Accreditation  Report  are 
analogous  to  final  T&E  reports  in  that 
they  aggregate  the  results  of  the  VV&A 
process. 

In  addition  to  the  Accreditation  Plan  and 
Accreditation  Report,  MCOTEA  also 
produces  V&V  Observation  Plans  and 
V&V  Observation  Reports,  explained 
further  in  this  chapter. 

Writing  the  Accreditation 
Plan 

The  Accreditation  Plan  is  drafted  early, 
typically  before  or  coincident  with  the 
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V&V  Plan,  but  is  intended  to  be  a  living 
document  that  can  be  adjusted  as  the 
M&S  and  W&A  process  progresses. 
The  ACA  should  plan  on  producing 
the  first  draft  of  the  Accreditation 
Plan  by  the  time  the  MCOTEA  SEP 
is  completed.  The  document  may  be 
MCOTEA-only  or  may  be  co-written 
with  the  PMO  if  the  M&S  will  be  used 
in  DT. 

From  MCOTEA’s  perspective  the  most 
important  element  of  the  Accreditation 
Plan  is  to  document  SIUs  and  define 
their  accreditation  criteria.  In  addition, 
the  Accreditation  Plan  defines  the 
methodology  for  conducting  the 
accreditation  assessment;  defines  the 
resources  needed  for  the  assessment;  and 
identifies  issues  or  concerns  associated 
with  performing  the  assessment. 

The  MCOTEA  COT  signs  the  plan 
when  complete  and  the  ACA  ensures 
that  the  Accreditation  Plan  is  sent  to  the 
DOD  M&S  Catalog,  and  is  entered  into 
the  MCOTEA  T&E  Reference  Center. 


Suitable  Government 
Representatives 

Normally,  the  ACA  or  another  member  of 
the  OT&E  team  will  witness  V&V  events. 
However,  if  attending  a  V&V  event  is  not 
practical,  MCOTEA  can  accept  V&V 
results  under  the  following  circumstances: 

♦  MCOTEA  receives  a  copy  of  the  event  plan 
before  the  event 

♦  The  event  is  witnessed  by  a  suitable 
government  representative  (can  be  a 
contractor  representing  the  government) 
familiar  with  the  M&S 

♦  The  government  representative  cannot  be 
employed  by,  or  subcontracted  to,  the  M&S 
Developer  or  the  system  development 
contractor  (if  the  M&S  supports  the  system 
under  test) 

♦  The  government  representative  records 
detailed  observations,  all  deviations  from 
the  plan,  and  all  caveats  associated  with  data 
elements 

♦  The  government  representative  is  available 
to  answer  MCOTEA  questions  after  the 
event 

♦  MCOTEA  has  access  to  all  recorded  event 
data 


Configuration  Management  Plan 

MCOTEA  requires  any  M&S  that  will 
undergo  changes  to  have  a  configuration 
management  plan.  Whenever  a  change  or 
a  group  of  changes  is  made  to  an  M&S, 
either  to  fix  errors  or  add  capabilities, 
the  version  of  the  M&S  changes.  The 
CMP  is  a  critical  component  of  the 
V&V  effort  because  it  is  essential  that  the 
version  that  undergoes  V&V  activities 
is  well  known  and  tightly  controlled. 
(Note:  a  V&V  activity  is  any  technique, 
analysis,  inspection,  demonstration,  or 
test  intended  to  verify  or  validate  the 
M&S.)  The  CMP  is  normally  written 
by  the  M&S  Developer  and  reviewed 
by  the  SCP.  The  CMP  exercises  control 
of  changes  to  the  M&S  and  supporting 
documentation  by  exercising  version 
control  and  tracking  code  changes.  It 
secures  the  code  against  unauthorized  or 


undocumented  changes,  and  provides  an 
audit  trail  of  all  changes  to  requirements 
and  the  M&S  all  the  way  back  to  original 
software  requirements.  A  good  CMP 
should  contain  software  status  accounting 
procedures,  procedures  for  managing 
changes  to  software  requirements,  control 
points  governing  scheduled  reviews,  as 
well  as  requirements  and  procedures  for 
regression  testing  when  changes  are  made 
to  the  M&S. 

As  part  of  good  configuration 
management,  the  following  should  be 
marked  with  the  appropriate  M&S  version 
number:  source  code,  executable  code, 
relevant  documentation,  input  data,  any 
special  hardware  associated  with  the  M&S, 
and  any  other  applicable  materials.  The 
MCOTEA  accreditation  process  applies 
to  everything  that  supports  the  specific 
version  of  M&S  that  will  support  OT&E. 
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♦  MCOTEA  receives  a  copy  of  all  reports 
generated  by  the  verification  team 
pertaining  to  the  verification  event 

The  MCOTEA  ACA  documents  the 
results  of  any  V&V  event  witnessed  by 
a  suitable  government  representative  in 
accordance  with  the  procedures  of  V&V 
Observation  Report. 

VEtV  Observation  Plan 

MCOTEA  requires  a  V&V  Observation 
Plan  (sample  p.  6-56)  before  any 
MCOTEA  representative  witnesses  an 
M&S  verification  or  validation  event. 
This  plan,  written  by  the  ACA  or  other 
MCOTEA  representative  witnessing  the 
V&V  event,  is  similar  to  the  Observation 
Plan  format  and  process  MCOTEA 
uses  for  DT  Observation.  The  V&V 
Observation  Plan  details  exactly  what  is 


being  verified  or  validated,  how  the  V&V 
event  is  expected  to  proceed,  and  describes 
the  anticipated  results  and  what  they 
mean.  Typically,  several  verification  and/or 
validation  techniques  or  activities  will  be 
scheduled  for  a  single  observation  event. 

The  ACA  should  obtain  the  plan  for  the 
V&V  event  as  soon  as  it  is  available.  The 
V&V  event  Plan  is  then  used  as  the  basis 
for  MCOTEA’s  Observation  Plan.  Each 
observation  requires  an  Observation  Plan, 
but  the  same  plan  can  be  used  to  observe 
multiple  V&V  events  close  together 
in  time.  The  ACA  submits  the  V&V 
Observation  Plan  to  the  COT  for  approval. 

VEtV  Observation  Report 

Even  though  V&V  tests  are  generally 
performed  by  other  entities,  the 
MCOTEA  ACA,  another  member  of  the 
OT&E  team,  or  a  suitable  government 


Table  6-3-2.  Outlines  of  Four  Core  VV& A  documents 


Accreditation  Plan  V&V  Plan  V&V  Report  Accreditation  Report 

Executive  Summary 

Executive  Summary 

Executive  Summary 

Executive  Summary 

1 .  Problem  Statement 

1 .  Problem  Statement 

1 .  Problem  Statement 

1 .  Problem  Statement 

2.  M&S  Requirements  and 
Acceptability  Criteria 

2.  M&S  Requirements  and 
Acceptability  Criteria 

2.  M&S  Requirements  and  Ac¬ 
ceptability  Criteria 

2.  M&S  Requirements  and  Accept¬ 
ability  Criteria 

3.  M&S  Assumptions,  Capa¬ 
bilities,  Limitations  &  Risks/ 
Impacts 

3.  M&S  Assumptions, 
Capabilities,  Limitations  & 
Risks/Impacts 

3.  M&S  Assumptions,  Capa¬ 
bilities,  Limitations  &  Risks/ 
Impacts 

3.  M&S  Assumptions,  Capabili¬ 
ties,  Limitations  &  Risks/Impacts 

4.  Accreditation  Methodology 

4.  V&V  Methodology 

4.  V&V  Task  Analysis 

4.  Accreditation  Assessment 

5.  Accreditation  Issues 

5.  V&V  Issues 

5.  V&V  Recommendations 

5.  Accreditation  Recommendations 

6.  Key  Participants 

6.  Key  Participants 

6.  Key  Participants 

6.  Key  Participants 

7.  Planned  Accreditation 

Resources 

7.  Planned  V&V  Resources 

7.  Actual  V&V  Resources 
Expended 

7.  Actual  Accreditation  Resources 

Expended 

8.  V&V  Lessons  Learned 

8.  Accreditation  Lessons  Learned 

Suggested  Annendices 

A.  M&S  Description 

B.  M&S  Requirements  Trace- 
ability  Matrix 

C.  Basis  of  Comparison 

D.  References 

E.  Acronyms 

F.  Glossary 

G.  Accreditation  Program¬ 
matics 

H.  Distribution  list 

Suggested  Annendices 

A.  M&S  Description 

B.  M&S  Requirements 
Traceability  Matrix 

C.  Basis  of  Comparison 

D.  References 

E.  Acronyms 

F.  Glossary 

G.  V&V  Programmatics 

H.  Distribution  list 

I.  Accreditation  Plan 

Suggested  Annendices 

A.  M&S  Description 

B.  M&S  Requirements  Trace- 
ability  Matrix 

C.  Basis  of  Comparison 

D.  References 

E.  Acronyms 

F.  Glossary 

G.  V&V  Programmatics 

H.  Distribution  List 

I.  V&V  Plan 

J.  Test  Information 

Suggested  Annendices 

A.  M&S  Description 

B.  M&S  Requirements  Traceabil¬ 
ity  Matrix 

C.  Basis  of  Comparison 

D.  References 

E.  Acronyms 

F.  Glossary 

G.  Accreditation  Programmatics 

H.  Distribution  List 

I.  Accreditation  Plan 

J.  V&V  Report 
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representative  must  be  present  to  certify 
the  results  of  the  observed  inspections, 
analyses,  demonstrations,  or  tests 
independently  from  those  conducting  the 
V&V  activities  and  in  their  own  words. 
After  returning  from  a  V&V  Observation, 
the  ACA  (or  the  actual  attendee)  writes 
the  V&V  Observation  Report  (sample  p. 
6-57),  which  records  the  outcome  of  all 
activities  for  the  observed  V&V  event  and 
the  extent  to  which  the  planned  V&V 
activities  were  executed. 

The  report  must  include  any  deviations 
from  the  plan,  any  activities  not 
performed,  or  any  activities  added  to  the 
original  event  plan.  The  V&V  Observation 
Report  documents  the  results  of  all  V&V 
activities  observed  from  the  MCOTEA 
perspective.  The  report  should  include 
all  relevant  test  plans,  any  relevant  data, 
and  the  results  of  the  testing,  if  known. 
MCOTEA  may  forward  the  Observation 
Report  to  the  M&S  Developer  and  M&S 
Proponent  after  COT  signature  if  the 
content  is  substantial  enough  that  the 
recipients  would  benefit  from  seeing  it. 
Otherwise  MCOTEA  retains  the  report 
internally  as  part  of  the  official  record  of 
W&A  activity. 

The  Observation  Reports  are  used  again 
towards  the  end  of  the  W&A  process 
after  MCOTEA  receives  the  official  V&V 
data  and  compares  the  record  of  observed 
events  with  the  V&V  Report. 

Typically,  the  V&V  Agent  rolls  up 
V&V  event  results  into  one  aggregated 
report  (the  V&V  Report),  meaning  that 
MCOTEA  will  most  likely  have  to  wait 
until  all  V&V  is  complete  before  receiving 
data  from  any  one  event;  however,  the 
ACA  should  contact  the  owner  of  the 
V&V  results  if  there  are  any  questions  on 
any  particular  V&V  event.  (MCOTEA 
may  also  request  data  along  the  way  if 
an  early  look  would  be  beneficial  to  the 
accreditation  process.) 

Once  the  V&V  Report  is  received,  the 
ACA  analyzes  the  data  and  results  for 


accuracy,  completeness,  and  for  fulfillment 
of  accreditation  criteria.  This  analysis  is 
included  in  the  Accreditation  Report. 

Accreditation  Report 

The  Accreditation  Report  is  typically 
written  by  the  ACA  and  summarizes 
all  data,  information,  and  activity, 
explicitly  or  by  reference,  used  in  the 
accreditation  assessment.  To  enable 
informed  accreditation  decisions,  the 
Accreditation  Report  must  provide  insight 
into  M&S  capabilities,  limitations,  and 
any  uncertainties  about  M&S  capabilities 
related  to  the  SIUs.  The  ACA  must  ensure 
that  the  following  information  is  accounted 
for  in  the  report  or  its  annexes: 

♦  Name  and  the  version  number  of  the  M&S 
being  accredited 

♦  Date  of  report  and  the  name/organization 
of  author  (accreditation  agent) 

♦  Description  of  the  M&S 

♦  Summary  of  model  assumptions 

♦  Summary  of  V&V  activities/processes 
performed  in  support  of  this  accreditation 

♦  Summary  of  previous  VV&A  activities  that 
apply  to  this  accreditation  and  why  they 
apply 

♦  Assessment  of  each  of  the  six  aspects  of  a 
MCOTEA  accreditation  as  explained  in  the 
Accreditation  Process  section  of  this  chapter 

External  references  and  documentation 
that  support  recommendations  in  the  report 
must  be  archived  in  the  MCOTEA  T&E 
Reference  Center,  regardless  of  who  wrote 
them. 

The  ACA  forwards  the  Accreditation 
Report  and  a  draft  Accreditation  Decision 
Letter  (explained  below)  to  the  MCOTEA 
CRB.  The  COT  signs  the  approved  report 
and  forwards  it  and  the  draft  Accreditation 
Decision  Letter  to  the  Accreditation 
Authority. 
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Accreditation  Decision  Letter 

The  ACA  drafts  the  Accreditation  Decision 
Letter  as  a  standard  naval  letter.  The  letter 
must  specify  M&S  name,  version  number, 
and  version  date  being  accredited.  The  letter’s 
content  is  based  on  the  recommendations 
resulting  from  the  Accreditation  Assessment, 
including  the  following: 

♦  The  degree  to  which  each  SIU  is  accredited 
(Fully,  Partially,  Decision  Pending,  or  Not 
Accredited) 

♦  Configuration  management  requirements  of 
the  M&S  in  order  to  maintain  accreditation 

♦  Any  requirements  for  the  data  used  as  input 
to  the  M&S  or  restrictions  on  the  data 
generated  by  the  M&S 

♦  Any  additional  V&V  requirements  by  SIU 

♦  Any  additional  questions  that  must  be 
answered  before  accreditation  by  SIU 


♦  Any  additional  documentation  required 
before  accreditation 

♦  A  description  of  any  limitations  on  the 
accreditation  decision 

The  Accrediation  Decision  Letter  remains 
in  effect  for  the  accredited  version  of  the 
M&S  as  long  as  the  intended  uses  remain 
unchanged,  or  until  revoked,  in  writing,  by 
the  AA. 

Accreditation  Decision 

The  MCOTEA  AA  has  the  following 
options  regarding  each  SIU: 

♦  Full  accreditation.  Fully  accredits  the  SIUs 
that  merit  full  accreditation. 

♦  Partial  accreditation.  SIUs  are  accredited 
under  certain  conditions  by  placing 
constraints  under  which  the  SIUs  may  be 
applied  to  OT&E. 

♦  Accreditation  Decision  Pending:  Full  SIU 


Separation  of  DT  and  OT&E 
Accreditations 

In  many  programs,  the  PMO  will  have  uses 
for  the  same  models  in  DT  that  MCOTEA 
intends  to  use  in  support  of  OT&E.  If  the 
PMO  intends  to  use  the  models  under 
consideration  for  DT  SIUs,  MCOTEA  can 
leverage  the  PMO’s  V&V  efforts.  Under 
these  circumstances  it  will  probably  make 
sense  for  the  DT  and  OT&E  accreditations 
to  use  the  same  Accreditation  Plan  since 
most  of  the  plans’  required  content  will  be 
the  same. 

Even  with  a  shared  plan,  the  SIUs  and 
accreditation  criteria  for  DT  and  OT&E  must 
be  called  out  separately  and  DT  and  OT&E 
SIU  accreditations  remain  completely  separate 
for  four  fundamental  reasons: 

•The  Accreditation  Authorities  for  DT  and 
OT&E  are  different 

•SIUs  for  DT  and  OT&E  are  independent 
of  one  another  and  most  likely  differ  from 
each  other 

•Different  validation  information  will  apply 
to  different  SIUs 


•DT  and  OT&E  timelines  are  different, 
and  accrediting  OT&E  M&S  later  than 
DT  M&S  may  allow  MCOTEA  to  take 
advantage  of  validation  opportunities  that 
might  arise  during  DT  and/or  OA  event 
execution. 

When  the  same  M&S  is  used  to  address 
both  DT  and  OT&E  issues,  MCOTEA 
works  closely  with  the  PMO  and  SCP 
to  resolve  any  issues  associated  with 
accreditation  to  increase  the  probability 
that  both  accreditations  can  be  successfully 
accomplished.  MCOTEA  OT&E 
team  members,  in  particular  the  ACA, 
should  strive  to  participate  in  all  of  the 
V&V  activities  associated  with  the  DT 
accreditation.  All  of  the  verification  activities 
associated  with  DT  accreditation  are  also 
required  by  the  MCOTEA  process,  and 
the  DT  validation  activities  will  be  useful 
in  building  MCOTEA  confidence  in  the 
M&S.  The  MCOTEA  ACA  must  ensure 
that,  in  addition  to  the  DT  V&V  activities, 
MCOTEA  requirements  for  verification  and 
validation  are  met. 
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accreditation  is  still  possible  assuming 
additional  information  is  received, 
additional  testing  is  accomplished,  or 
modifications  are  made  to  the  M&S. 

♦  Not  accredited:  The  SIU  cannot  be 
accredited  to  support  OT&E. 

The  different  accreditation  options  apply  by 
SIU.  Therefore,  an  Accreditation  Decision 
could  conceivably  fully  accredit  some  SIUs, 
partially  accredit  others,  conditionally 
accredit  some,  and  not  accredit  still  others. 

Partially  accredited  SIUs,  conditional 
SIUs  that  do  not  undergo  remediation,  or 
unaccredited  SIUs  imply  limitations  to  the 
OT&E. 

The  MCOTEA  AA  signs  the  letter 
after  making  any  desired  changes.  The 
accreditation  is  not  official  until  the  letter 
is  signed.  The  letter  remains  in  effect  for 
the  accredited  version  of  the  model  as  long 
as  the  intended  uses  remain  unchanged,  or 
until  revoked  by  the  AA. 

The  ACA  is  responsible  for  filing  the 
signed  Accreditation  Decision  Letter  in 
the  MCOTEA  T&E  Reference  Center 
and  the  DOD  M&S  Catalog. 

Accounting  for  Previous 
Accrediation 

MCOTEA  strives  to  leverage  any  previous 
W&A  activity  for  the  model  under 
consideration  to  the  maximum  extent 
possible,  but  MCOTEA  determines  its 
V&V  requirements  independently  of 
what  has  already  been  accomplished. 

This  independent  examination  of  V&V 
requirements  may  result  in  the  need  for 
additional  V&V  activities. 

MCOTEA  may  reuse  any  unaltered 
M&S  version  previously  accredited 
by  MCOTEA  for  a  given  set  of  SIUs 
assuming  the  previous  accreditation  criteria 
are  acceptable  for  the  new  application. 
However,  if  the  M&S  has  been  modified 
in  some  way,  the  SIUs  are  different,  or  the 
accreditation  criteria  have  changed,  a  new 
accreditation  is  required. 


Following  are  four  examples  of  situations 
that  require  new  accreditation  but  can 
most  likely  accept  previous  V&V  or 
portions  of  it: 

Modified  MEtS  Version 

Situation  1:  MCOTEA  has  accredited 
M&S  version  1.0  and  later  would  like 
to  modify  it  and  use  version  1.1  to 
support  MCOTEA  testing  or  evaluation. 
Response:  MCOTEA  must  separately 
accredit  version  1.1.  In  this  case,  at  least 
some  of  the  original  V&V  work  is  likely 
to  be  usable  in  support  of  version  1.1 

V&V. 

Same  MEtS,  Different  SIUs 

Situation  2:  MCOTEA  has  accredited 
M&S  version  1.0  for  SIUs  for  a  particular 
application  and  would  like  to  reuse  this 
version  for  different  SIUs.  For  example, 
MCOTEA  may  have  accredited  M&S 
in  support  of  the  OT  of  a  chem-bio 
protective  garment  that  models  chemical 
penetration  of  the  garment  and  chemical 
burns  to  the  wearer.  Later  use  might  be 
to  supplement  the  OT  of  a  non-lethal 
weapon  system  by  modeling  burns  from 
heat  sources.  Response:  Version  1.0  must 
be  reaccredited  because  the  thermal  burn 
SIUs  must  be  accredited  separately  from 
the  chemical  burn  SIUs.  Presumably, 
however,  most  of  the  original  verification 
efforts  and  perhaps  some  of  the  original 
validation  efforts  could  be  reused  in  the 
second  accreditation. 

Same  M&S  and  SIUs,  Different 
Accreditation  Criteria 

Situation  3:  MCOTEA  has  accredited 
M&S  version  1.0  SIUs  for  one  test 
article  and  would  like  to  reuse  this  same 
M&S  version  and  SIUs  for  a  different 
but  related  test  article  (chem-bio 
garments,  for  example).  Assuming  that 
the  accreditation  criteria  supporting 
the  OT&E  of  the  first  garment  are 
different  from  the  criteria  for  the 
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second  OT&E,  MCOTEA  must  accredit 
the  M&S  separately  to  satisfy  the  new 
criteria.  However,  most  of  the  work  from 
the  previous  W&A  effort  presumably 
could  be  reused,  leaving  only  minor 
validation  activities  required  for  the  second 
accreditation. 

M&S  Accreditation  from  Another 
Organization 

Situation  4:  Another  organization 
accredits  an  M&S  with  exactly  the  same 
SIUs  that  MCOTEA  needs  the  model 
to  support.  Response:  MCOTEA  must 
still  independently  accredit  the  M&S  to 
support  OT&E,  even  if  the  previously 
accredited  version  of  the  M&S  is  identical 
to  the  one  MCOTEA  wants  to  use.  The 
first  reason  for  this  is  directive  in  nature: 
only  MCOTEA  can  accredit  an  M&S 
for  use  in  a  MCOTEA  OT&E.  In 
addition  (and  fundamental  to  the  concept 
of  accreditation),  no  guarantee  exists  that 
the  other  organization’s  accreditation 
process  meets  MCOTEA’s  accreditation 
requirements.  In  summary,  extensive 
previous  use  of  an  M&S  or  accreditation  by 
another  organization  does  not  automatically 
guarantee  accreditation  of  the  M&S  for 
SIUs  in  support  of  MCOTEA  OT&E. 

See  the  section  below  for  details  on 
reaccreditation. 

MCOTEA’s  Reaccreditation 
Process 

“Any  subsequent  use  in  a  new  application 
domain  or  modification  of  the  M&S  will 
require  a  reaccreditation  process”  (SNI 
5200.40).  The  MCOTEA  reaccreditation 
process  is  the  same  as  the  accreditation 
process,  except  that  the  ACA  will  leverage 
as  much  of  the  V&V  efforts  from  any 
previous  accreditations  as  possible. 

The  degree  to  which  the  information  from 
any  previous  W&A  effort  can  be  reused 
depends  on  the  quality  of  the  associated 
documentation.  The  ACA  must  be  able  to 
discern  the  following  elements  of  quality  in 
W&A  documentation: 


♦  The  exact  version  of  the  M&S  previously 
accredited  must  be  evident. 

♦  The  M&S  must  not  have  changed,  or 
the  change  and  regression  testing  of 
the  changed  M&S  must  be  sufficiently 
documented. 

♦  Terms  such  as  “accurate,” “sufficient,” 
or  “adequate”  must  be  supported  by 
documented  evidence. 

♦  The  documentation  must  clearly  discuss 
W&A  procedures  and  data  and  the  results 
of  inspections,  analyses,  demonstrations, 
and  tests. 

♦  The  details  of  a  V&V  event  should  include 
exactly  what  was  done  and  under  what 
conditions,  who  observed  and  documented 
the  event,  the  resulting  data,  how  the  data 
was  analyzed,  and  the  factual  results  of  the 
analyses. 

Where  to  start 

The  ACA  begins  the  reaccreditation  process 
by  following  the  same  steps  used  for  initial 
accreditation.  Therefore,  the  ACA  needs  to 
examine  the  following  basic  information: 

♦  The  M&S  Developer’s  software  development 
and  software  quality  assurance  processes 

♦  What  the  M&S  does  and  how  it  does  it 

♦  The  basic  assumptions  used  in  the  M&S 

♦  Conceptual  model 

♦  User’s  Manual 

♦  Programmer’s  manual 

♦  Any  other  available  introductory 
documentation 

♦  Documents  that  describe  past  actions 

♦  PRevious  Accreditation  Plans 

♦  Previous  V&V  Reports 

♦  Previous  Accreditation  Reports 

♦  Configuration  Management  documentation 

The  Previous  Accreditation  Plan  will 
show  what  was  intended  in  the  previous 
accreditation  and  the  Report  will  show  what 
was  actually  accomplished.  The  Previous 
V&V  Report  should  contain  a  wealth  of 
information  in  support  of  the  accreditation. 
If  elements  of  the  Accreditation  Plan  and 
V&V  Report  are  not  addressed  in  the 
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Accreditation  Report,  the  ACA  needs 
to  understand  why  this  is  the  case.  The 
Accreditation  Report  should  also  include 
several  references,  typically  sources  of 
data  or  documented  tests  used  to  support 
the  accreditation. 

For  MCOTEA  to  accept  previous 
accreditation  results,  the  M&S  must  have 
been  under  strict  configuration  control 
between  the  previous  accreditation  and 
the  present.  If  the  M&S  version  has 
changed  in  any  way  since  the  previous 
accreditation  but  no  record  exists  of  the 
changes  or  of  V&V  to  support  those 
changes,  the  ACA  must  plan  appropriate 
V&V  activities  to  compensate  for  this 
shortfall. 

Using  Previous  Verification 
& Validation  Efforts 

Previous  Verification 

MCOTEA’s  accreditation  requirements 
for  verification  remain  the  same  for 
first-time  verification  and  in  support  of 
reaccreditation. 

If  the  model  of  current  interest  to 
MCOTEA  has  changed  from  the 
original,  verified  version,  the  ACA  can 
still  use  the  previous  W&A  information 
to  gain  familiarity  with  the  model’s 
capabilities.  Although  the  code  itself  will 
have  changed  from  version  to  version, 
functional  modules  within  the  code  may 
or  may  not  have  changed.  To  the  extent 
that  functional  modules  have  not  changed 
from  the  original  version,  the  verification 
efforts  of  those  modules  may  still  be 
applicable.  However,  those  efforts  may  yet 
be  insufficient  to  meet  MCOTEA’s  needs. 
Depending  on  the  thoroughness  of  the 
previous  verification  effort,  MCOTEA 
may  require  additional  verification  of 
codes  that  have  already  undergone  a  set  of 
verification  procedures. 

In  any  case,  when  the  previously  verified 
model  version  has  changed,  all  modified 


functional  modules  of  that  version  and  the 
interactions  between  all  modules  need  to 
be  re-verified.  If  changes  to  the  M&S  were 
not  sufficiently  documented,  the  entire  code 
will  require  some  level  of  new  verification 
activity.  Under  these  circumstances  some 
of  the  verification  techniques  described  in 
this  chapter,  such  as  modular  string  testing, 
should  be  considered. 

The  ACA  must  document  all  previous 
verification  activities  used  in  the  MCOTEA 
accredidation  in  the  Accreditation  Report, 
along  with  any  supplemental  verification 
activities  required  by  MCOTEA. 

Previous  Validation 

Past  successful  validation  efforts  should 
give  the  ACA  a  degree  of  confidence  in  the 
M&S.  However,  unlike  certain  verification 
techniques  that  focus  on  functional 
modules  of  the  M&S,  validation  testing 
tends  to  examine  the  validity  of  the  overall 
M&S.  Therefore,  if  the  M&S  has  changed 
at  all  since  the  last  accreditation,  previous 
validation  efforts  relevant  to  the  current 
SIUs  may  need  to  be  repeated  and  new 
validation  activities  may  be  required.  At  a 
minimum  this  will  involve  ensuring  that 
the  M&S  meets  the  accreditation  criteria 
in  the  new  Accreditation  Plan. 

Independent  Verification 
and  Validation 

Independent  Verification  and  Validation 
(IV&V)  is,  by  definition,  independent  of  the 
M&S  Proponent  and  the  M&S  Developer’s 
regular  V&V  of  a  model.  IV&V  is  optional 
unless  directed  by  the  M&S  Proponent, 
the  AA,  or  a  higher  authority.  MCOTEA 
may  direct  that  an  IV&V  be  conducted  on 
an  M&S  if  the  MCOTEA  AA  believes 
it  is  necessary  to  establish  the  requisite 
level  of  confidence  in  a  model  for  support 
of  OT&E.  The  requirements  for  IV&V 
are  identical  to  those  of  a  regular  V&V  as 
described  in  this  manual;  the  only  difference 
is  the  entity  performing  the  V&V. 
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Results  of  the  I  V&V  are  documented  in 
a  separate  IV&V  Report,  analogous  to  a 
V&V  Report,  and  should  contain  all  of  the 
V&V  Report  elements.  The  report  should 
be  delivered  to  the  ACA  for  consideration 
and  analysis  in  conjunction  with  other 
material  in  support  of  the  accreditation. 


V&V  Observation  Plan 

1.  Purpose.  [State  the  purpose  of  this  document,  the  purpose  of  the  event,  and  its  date 
and  location. 

2.  Background.  [State  the  program  the  M&S  is  meant  to  support  and  why  this  V&V 
is  taking  place.  Describe  how  M&S  accuracy  will  be  verified,  what  M&S  requirements 
will  be  verified,  and  what  is  being  validated.] 

3.  Schedule.  [State  the  schedule  and  sequence  of  expected  V&V  activities.] 

4.  Organization.  [State  the  expected  event  participants  and  the  MCOTEA  observation 
team  (by  name)  and  describe  their  function  during  the  V&V  event.] 

5.  Evaluation  Questions.  [Describe  the  expected  V&V  techniques  or  activities 
and  connect  the  event  to  the  V&V  Plan.  Note  the  parts  of  the  V&V  Plan  and  the 
MCOTEA  Accreditation  Plan  satisfied  by  each  V&V  activity.  Describe  what  the 
observer  expects  to  see  for  each  activity  and  convey  an  understanding  of  how  each 
V&V  activity  contributes  to  MCOTEA’s  level  of  confidence  in  the  M&S.  Note  the 
significance  of  the  expected  data  and  observations  for  each  V&V  activity  and  consider 
the  implications  of  alternative  V&V  outcomes.  ] 

6.  References 

MCOTEA.  (M&S  Title)  Accreditation  Plan.  [Month  Year], 

(Other  references  as  required) 

Annex  A.  V&V  Test  Plans  associated  with  this  V&V  event. 
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V&V  Observation  Report 

1.  Purpose.  [State  the  purpose  of  this 
document  (to  provide  MCOTEA’s 
observations  and  assessment  of  V&V 
event  execution)  and  the  purpose, 
location,  and  date  of  the  V&V  event.] 

2.  Background.  [Restate  the 
background  section  of  the  V&V 
Observation  Plan.] 

3.  Scope.  This  report  documents 
MCOTEA’s  observations  of  [event] 
execution.  Summarize  all  V&V 
activities  that  actually  took  place  during 
this  V&V  event.  This  will  lay  the 
groundwork  for  addressing  them  all  later 
in  the  report.  Conclusions  may  be  drawn 
if  sufficient  information  is  available. 

4.  Objective.  The  objective  of  this 
report  is  to  formally  record  MCOTEA’s 
observations  of  test  execution  before 
receiving  the  V&V  Report  and  to 
independently  certify  the  results  of 
V&V  activities  requiring  no  further 
analysis. 

5.  Assumptions.  [List  any  assumptions 
made  during  any  V&V  activity. 
Otherwise,  N/A.] 

6.  Limitations.  [List  V&V  activities 
scheduled  for  this  V&V  event  but  that 
did  not  take  place  and  the  reason  for 
their  not  occurring.  Also  list  V&V 
activities  for  which  there  was  insufficient 
information,  MCOTEA  is  awaiting 
data,  or  for  any  other  reason  MCOTEA 
could  not  make  a  certification  statement 
about  the  V&V  activity  at  the  time  of 
this  report.] 

7.  Methods.  [Cite  the  method 
MCOTEA  used  to  certify  each  V&V 
activity,  e.g.,  observed  modular  string 
testing,  followed  the  steps  of  an  SME 
analysis,  etc.] 

8.  Results.  [List  all  V&V  activities  that 
MCOTEA  is  able  to  certify  met  the 
criteria  for  verification  or  validation. 

Also  list  any  V&V  activities  that 


MCOTEA  is  able  to  independently 
certify  did  not  meet  the  verification  or 
validation  criteria  based  on  this  V&V 
event.] 

9.  Insights.  [Preface  any  statements 
here  with  “It  appears  that.”  Address 
any  strengths  or  deficiencies  in  the 
M&S  that  became  apparent  during  the 
V&V  event.  Also  list  any  new  M&S 
assumptions  that  became  apparent.  If 
the  event  did  not  yield  any  Insights, 
use  N/A.] 

10.  Recommendations  [Recommend 
any  additional  V&V  activities  that 
should  take  place  for  the  parts  of  the 
M&S  addressed  during  this  event.  If 
recommending  further  testing,  submit 
this  report  to  the  M&S  Proponent.] 

11.  References 

a.  MCOTEA.  [V&V  event] 
Observation  Plan.  [Month  Year], 

b.  [Author].  [Applicable  V&V  test 
plan],  [Month  Year]. 


6-57 


Chapter  6 


Further  Information  About  VElV  Documentation 
Verification  and  Validation  Plan 

MCOTEA  requires  a  V&V  Plan,  in  accordance  with  SNI  5200.40,  for  any  M&S  that 
requires  additional  verification,  validation,  or  both.  The  V&V  Plan  is  developed  by  the  M&S 
Developer  or  the  M&S  Proponent,  with  MCOTEA  input.  The  ACA  and  SCP  should 
review  the  plan  for  accuracy  and  completeness.  The  SCP  must  specify  the  due  date  of  the 
V&V  Plan  based  on  realistic  estimates  for  model  completion  and  program  schedule.  The 
V&V  Plan  is  a  living  document,  adjusted  as  the  M&S  and  W&A  processes  progress. 

The  contents  of  the  V&V  Plan  are  seen  in  figure  XX.  Ideally  the  V&V  Plan  includes  the 
test  plans  for  all  V&V  activities  that  require  testing;  however,  this  level  of  detail  may  be 
filled  in  later.  MCOTEA  must  receive  a  copy  of  the  detailed  test  plan  at  least  15  days  before 
any  V&V  event. 

For  legacy  models  (modified  or  requiring  additional  V&V  activities),  the  V&V  Plan 
addresses  legacy  model  assumptions,  capabilities,  and  any  previous  VV&A  activities  as 
well  as  an  explanation  of  all  planned  M&S  enhancements  and  all  planned  V&V  activities. 
Although  MCOTEA  leverages  all  previous,  relevant  V&V  activities,  MCOTEA 
determines  its  V&V  requirements  independent  of  what  has  already  been  accomplished. 

If  the  previous  V&V  efforts  were  insufficient  or  undocumented,  MCOTEA  may  require 
additional  V&V.  In  addition,  MCOTEA  will  still  require  that  the  M&S  satisfies  the 
accreditation  criteria  for  OT&E  SIUs. 

Verification  and  Validation  Report 

MCOTEA  requires  a  V&V  Report,  in  accordance  with  SNI  5200.40,  to  document 
and  describe  the  details  of  all  V&V  events.  The  V&V  Report  is  developed  by  the  M&S 
Developer  or  the  M&S  Proponent,  with  MCOTEA  input.  This  report  documents 
evidence  supporting  the  functionality  and  fidelity  of  M&S  to  satisfy  OT&E  SIUs,  M&S 
requirements,  and  model  accuracy  requirements.  The  V&V  Report  documents  the  M&S 
Developer,  the  M&S  Description,  M&S  assumptions,  and  any  risks  associated  with  using 
the  M&S  or  associated  data.  The  V&V  Report  details  all  verification  and  validation  activity 
to  include 

♦  a  complete  description  of  V&V  methodologies,  organizations,  and  individuals  involved  in  V&V 
and  a  summary  of  their  findings 

♦  a  description  of  actions  taken  as  a  result  of  V&V 

♦  explicit  identification  of  known  M&S  capabilities,  limitations,  and  restrictions 

♦  detailed  descriptions  of  all  V&V  techniques,  analyses,  inspections,  demonstrations,  and  tests  to 
include  scope,  limitations,  methodology,  scenarios,  environments,  participants,  and  all  supporting 
data 

♦  a  compilation  of  any  V&V  reports  pertaining  to  previous  relevant  V&V  activities  being  leveraged 
for  the  current  V&V  effort 

♦  data  V&V  activities  including  the  original  reason  the  data  was  generated,  how  the  data  was 
generated  (the  more  detail  the  better),  and  any  assumptions  made  in  generating  the  data 

The  V&V  Report  should  be  designed  for  use  as  a  reference  for  follow-on  W&A  activities 
and  for  future  regression  testing. 
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Annex  A.  V8tV  Process  and  Techniques 


Basics  of  Verification 

Before  discussing  how  to  verify  something, 
it  is  useful  to  repeat  the  definition  of 
verification.  According  to  reference  (DODI 
5000.61),  verification  is,  “The  process  of 
determining  that  a  model  or  simulation 
implementation  and  its  associated  data 
accurately  represent  the  developer’s 
conceptual  description  and  specifications.” 

The  bolded  words  indicate  the  two  aspects 
of  verification.  The  first  is  accuracy.  For 
example,  if  the  model  is  a  computer  code, 
one  must  acknowledge  there  are  always 
undetected  errors  in  the  code;  the  larger 
the  code,  the  more  undetected  errors. 
(Anybody  doubting  this  assertion  should 
consult  Microsoft  about  the  “Blue  Screen 
of  Death”.)  The  goal  is  to  minimize  the 
number  of  undetected  errors,  thus  ensuring 
the  code  is  “accurate”. 

The  second  aspect  of  verification  is  to 
ensure  the  code  reflects  the  specifications 
spelled  out  for  its  construction  (normally  in 
a  Software  Requirements  Specification).  If 
the  code  doesn’t  do  what  it  was  supposed 
to  do,  it  doesn’t  matter  how  accurately 
it  does  it.  Typically,  model  developers 
will  emphasize  this  aspect  of  verification 
because  it  is  easy  to  list  requirements  and 
show  how  they  will  be  verified.  Checking 
the  M&S  for  accuracy  is  arguably  more 
difficult. 

At  this  point  it  is  useful  to  note  that 
checking  an  M&S  against  its  requirements 
is  typically  a  verification  function. 
Occasionally,  a  requirement  will  spell 
out  an  M&S  capability  as  compared  to 
the  corresponding  real  world  capability 
(resulting  in  a  validation  of  that 
requirement),  but  this  is  rare.  Requirements 
are  typically  “verified”,  while  validation 
is  used  to  confirm  the  M&S  is  a  realistic 
representation  of  the  real  world  and  is 
capable  of  satisfying  the  designated  specific 
intended  uses. 

The  remainder  of  this  section  describes 


selected  verification  techniques.  The  DOD 
VV&A  Recommended  Practices  Guide 
contains  several  additional  verification 
techniques  that  should  be  considered.  The 
ACA  should  coordinate  with  the  V&V 
Agent/SCP  and  the  Model  Developer  to 
determine  the  set  of  verification  procedures 
that  makes  the  most  sense  for  the  M&S 
under  consideration. 

It  is  extremely  important  that  all 
techniques  used  to  verify  the  M&S 
be  thoroughly  documented  in  the 
V&V  Report,  and  summarized  in  the 
Accreditation  Report.  This  increases  the 
credibility  of  both  reports  and  allows  for 
reuse  of  V&V  work  in  future  accreditation 
efforts. 

Verifying  an  M&S  for  Accuracy 

As  discussed,  the  object  is  to  minimize 
the  number  of  undetected  errors  in  the 
M&S.  When  a  code  is  first  being  written, 
if  there  are  errors  in  implementing  the 
computer  language,  the  code  will  generally 
not  run  until  these  “syntax”  errors  are 
corrected.  These  are  the  easy  errors  to  track 
down,  and  an  experienced  programmer 
can  frequently  construct  a  section  of  code 
without  any  errors  in  syntax. 

It  is  the  errors  in  logic  that  are  the  most 
difficult  to  detect  and  locate.  When  a 
software  engineer  constructs  a  code,  he/ 
she  invariably  is  required  to  make  certain, 
seemingly  benign  assumptions.  Since  it 
is  a  rarity  for  the  software  engineer  to  be 
a  SME  in  the  real  world  processes  being 
modeled,  these  assumptions  are  sometimes 
erroneous.  It  is  wise  to  have  SMEs 
available  to  answer  the  questions  of  the 
software  engineers  while  coding  the  M&S; 
however,  even  if  an  SME  is  available  to 
answer  questions  during  coding,  erroneous 
assumptions  and  other  errors  in  logic 
can  still  be  implemented  in  the  code. 

The  following  describe  some  techniques 
useful  in  understanding  the  M&S  and 
locating  errors  within  it.  Other  verification 
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techniques  can  be  found  in  reference 
(DOD  VV&A  Recommended  Practices 
Guide,  2001). 

Another  aspect  of  verifying  a  model  for 
accuracy  is  the  examination  of  how  the 
M&S  accommodates  unanticipated  or  out 
of  specification  range  inputs.  The  model 
should  be  protected  from  erroneous  data 
entry.  Furthermore,  models  should  not 
allow  representations  that  violate  the  laws 
of  physics. 

Documentation  Walkthrough 

It  is  important  to  identify  the  assumptions 
made  when  the  model  was  coded.  This 
helps  in  determining  whether  a  model 
is  appropriate  or  not  for  a  specific  use. 

One  way  to  identify  these  assumptions  is 
to  systematically  go  through  the  model 
documentation.  Many  of  the  explicit 
assumptions  made  in  the  construction  of 
the  M&S,  its  internal  parameters,  or  other 
input  data  can  be  determined  by  a  careful 
review  of  the  M&S  User’s  Manual  or  any 
other  documentation  that  describes  the 
logic  used  in  the  M&S.  Ideally,  this  review 
would  be  accomplished  by  the  appropriate 
SME. 

In  some  cases,  the  M&S  documentation 
required  for  the  Documentation 
Walkthrough  will  not  exist.  In  those 
cases,  the  verification  effort  is  obviously 
weakened  and  the  explicit  assumptions  will 
have  to  be  identified  by  interviewing  the 
software  engineers  who  wrote  the  M&S 
and  by  inspection  of  the  M&S  itself. 

MEtS  Inspection 

The  source  code  should  also  be  checked  to 
see  exactly  how  the  explicit  assumptions 
are  implemented.  The  inspection  of 
each  functional  module  (subroutine, 
object,  etc)  is  accomplished  by  software 
engineers  conversant  in  the  computer 
language  used  in  constructing  the  M&S. 

It  can  be  performed  by  employees  of  the 
M&S  Developer,  but  it  is  preferable  this 
be  performed  by  software  engineers  not 


involved  in  coding  the  M&S. 

Inspection  serves  three  purposes.  First,  as 
the  software  engineer  goes  through  the 
module,  he/she  makes  note  of  how  all 
explicit  assumptions  were  implemented 
in  the  model.  In  addition,  this  inspection 
can  be  used  to  identify  assumptions 
implicit  in  the  way  the  model  itself 
was  coded,  including  noting  any  fixed 
parameters  coded  into  the  M&S.  These 
implicit  assumptions  should  be  checked 
with  the  appropriate  SME  for  accuracy. 
Lastly,  the  inspection  of  each  functional 
module  also  generally  represents  the  first 
time  independent  experts  have  had  an 
opportunity  to  locate  logic  errors  in  the 
M&S.  Again,  the  presence  of  a  SME  will 
facilitate  finding  the  logic  errors  at  this 
stage. 

It  would  be  ideal  if  the  SME  that  examines 
the  documentation  and  the  software 
engineer  that  examines  the  model  are  the 
same  person,  but  it  is  rare  to  find  these 
diverse  capabilities  in  one  person.  These 
operational  and  software  engineering 
reviewers  can  be  part  of  the  DT  team, 
members  of  the  SCP,  independent 
contractors,  or  employees  of  the  software 
developer,  but  they  should  not  be  the  same 
people  that  developed  the  M&S  in  the 
first  place.  The  fact  that  the  inspection(s) 
was  performed,  how  it  was  performed,  and 
the  results  of  the  inspection,  to  include 
a  review  of  all  identified  assumptions 
and  any  errors  that  were  discovered, 
should  be  documented  in  the  V&V 
report.  Additional  information  on  the 
formal  inspection  process  can  be  found  in 
reference  (DOD  W&A  Recommended 
Practices  Guide). 

Modular  String  Testing 

Before  checking  strings  of  modules, 
individual  modules  (subroutine,  object, 
etc.)  should  be  checked  for  correct  and 
accurate  behavior.  When  checking  the 
behavior  of  individual  modules,  it  is  often 
worthwhile  to  “instrument”  the  module, 
that  is,  insert  additional  code  in  the 
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M&S  in  order  to  record  parameter  values 
at  strategic  points  in  the  module.  This 
instrumentation  of  the  module  allows  for 
tracking  parameter  values  to  ensure  the 
module  behaves  as  expected. 

After  testing  functional  modules 
individually,  it  is  useful  for  the  M&S 
Developer  to  test  strings  of  modules  to 
ensure  interface  logic  between  modules 
and  model  outputs  are  consistent  with 
expectations  (fig.  6-3-4).  First,  the  software 
engineer  decides  on  a  logical  grouping  of 
modules  to  test.  After  constructing  the 
necessary  input  data,  the  software  engineer 
does  a  hand  calculation  on  the  expected 
outputs,  based  on  his/her  understanding 
of  each  functional  software  module  being 
tested.  The  inputs,  modules  being  tested 
and  expected  outputs  are  all  documented 
in  the  V&V  plan  and  V&V  report.  The 
results  of  each  test  run  are  used  to  locate 
any  logical  errors  in  the  modules  under  test. 
Some  notional  examples  of  this  technique 
are  shown  in  figure  C.  Some  modules  may 
be  tested  more  than  once,  but  all  should 
be  tested  at  least  once.  Furthermore,  the 
paths  through  the  M&S  that  will  be 
frequently  used  should  also  be  tested  in  this 
way.  Note  that  this  technique  only  tests 
operational  logic  that  the  software  engineer 
understands.  Here  again  it  is  helpful  to 
enlist  an  operational  or  system  SME  in 
order  to  capture  as  many  logical  errors  as 
possible  in  the  M&S.  Instrumentation  of 
the  modular  string  is  also  useful  in  locating 
errors  and  confirming  desired  results. 

From  MCOTEA’s  perspective,  the  OT&E 
team’s  ACA  only  needs  to  see  each  modular 
string  test  performed  once  -  correctly.  This 
allows  the  M&S  Developer  to  conduct 
the  test  as  many  times  as  needed  to  catch 
all  the  logic  errors  the  test  is  capable  of 
catching  before  MCOTEA  verifies  the 
test  has  been  successfully  completed.  The 
modular  string  testing,  including  the  input 
data,  the  modules  tested  and  the  output 
attained  should  be  reported  in  the  V&V 
report  and  the  Accreditation  Report. 


Known 
inputs  #4 


0=  Functional  Software  Module 


Fig.  6-3-4. 
Testing  Strings 
of  Software 
Modules 


A  big  advantage  to  this  technique  applies 
to  regression  testing  of  the  M&S  when  the 
model  is  changed  because  errors  are 
corrected  in  the  version  of  interest,  or  when 
verifying  a  follow-on  version  of  the  model 
in  a  future  verification  effort.  Since  these 
test  cases  are  thoroughly  documented  in 
the  V&V  plan,  V&V  report,  and 
Accreditation  Report,  it  should  be 
relatively  easy  to  repeat  the  testing,  as 
required,  to  ensure  no  unwanted  changes  in 
M&S  behavior  have  been  introduced  by 
changes  to  the  M&S  due  to  error 
correction  or  changes  in  model 
functionality.  If,  during  the  verification  and 
validation  process,  changes  are  made  to  a 
functional  software  module  that  are 
designed  to  fix  newly  discovered  errors,  at  a 
minimum  all  verification  tests  that  involve 
that  module  must  be  re-run  on  the  final 
version  of  the  model  (the  version  to  be 
used  during  OT&E).  As  a  best  practice,  all 
documented  verification  tests,  regardless  of 
whether  the  included  functional  software 
modules  have  been  modified  or  not,  should 
be  re-run  on  the  final  version  of  the  model. 

Verifying  That  an  M&S  Meets 
Specifications 

Typically  the  expectations  for  an  M&S  are 
spelled  out  in  the  Software  Requirements 
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Specification  (SRS).  Each  of  the 
requirements  should  be  called  out  in  the 
V&V  plan.  Verification  of  the  requirements 
is  typically  done  by  Inspection,  Analysis, 
Demonstration,  or  Test.  The  choice  of 
verification  methods  to  use  on  a  particular 
requirement  is  left  to  the  organization 
doing  the  verification  (probably  the  M&S 
Developer)  with  concurrence  of  the  SCP. 
The  verification  methods  are  described 
below: 

Inspection:  The  examination  and  review  of 
descriptive  documentation  and  comparison 
of  appropriate  characteristics  with  a 
predetermined  standard.  This  method  may 
require  access  to  the  source  code. 

Analysis:  Analysis  includes  quantitative 
and/or  qualitative  proof  that  the  code 
meets  specific  requirements  by  technical 
evaluation  using  mathematical  equations, 
charts,  graphs,  and  representative  data. 

Demonstration:  This  involves  the  operation 
or  adjustment  of  the  code.  The  code  may 
be  instrumented  and  its  performance 
monitored,  but  only  as  an  indirect 
function  in  support  of  the  demonstration. 
Quantitative  measurements  are  generally 
not  taken  except  in  cases  where  test 
operators  make  visual  measurements/ 
counts  or  where  simple  devices  such  as 
a  stopwatch  are  used  to  estimate  time 
performance.  Generally,  demonstration 
results  may  be  noted  by  a  simple  YES 
or  NO.  Success  and  failure  criteria  will 
be  established  for  each  demonstration 
objective  prior  to  the  demonstration. 

Test:  Exercising  the  applicable  code 
under  appropriate  conditions  with 
instrumentation  to  collect/analyze/evaluate 
the  data  to  ensure  the  requirements  are 
met.  Acceptability  of  the  code  will  be 
determined  by  pre-established  quantitative 
criteria  consistent  with  the  required 
characteristics  stated  in  the  applicable 
specification.  A  test  plan  is  generated 
before  each  test. 

Just  as  in  the  case  with  verifying  the  M&S 


for  accuracy,  when  verifying  that  the 
M&S  meets  its  requirements,  MCOTEA 
only  needs  to  see  results  when  the  M&S 
Developer  is  comfortable  the  verification 
will  be  a  success.  MCOTEA  requirements 
for  the  four  verification  methods  are: 

Inspection:  MCOTEA  receives  a  plan 
describing  what  is  to  be  inspected  and 
how  it  will  be  inspected  before  the 
verification  event.  A  member  of  the 
MCOTEA  OT&E  team  or  a  suitable 
government  representative  is  present 
during  the  inspection,  can  ask  questions 
during  the  inspection  and  answer  future 
questions  about  the  inspection,  and  can 
independently  confirm  the  inspection 
results. 

Analysis:  MCOTEA  receives  a  copy  of  the 
full  analysis  and  any  associated  assumptions 
and  data.  MCOTEA  must  have  access 
to  those  that  did  the  analysis  to  answer 
questions.  A  member  of  the  MCOTEA 
OT&E  team  must  independently  confirm 
that  the  analysis  is  correct. 

Demonstration:  MCOTEA  receives  a  plan 
describing  the  demonstration,  including 
what  is  to  be  demonstrated.  A  member  of 
the  MCOTEA  OT&E  team  or  a  suitable 
government  representative  is  present 
during  the  demonstration,  can  answer 
future  questions  about  the  demonstration, 
and  can  independently  confirm  the 
demonstration  results. 

Test:  MCOTEA  receives  a  copy  of  the 
test  plan  for  review  and  comment.  A 
member  of  the  MCOTEA  OT&E  team 
or  a  suitable  government  representative 
is  present  during  the  test,  can  answer 
questions  about  the  test,  and  can 
independently  confirm  the  test  results. 

Generally  speaking,  MCOTEA  will  want 
the  appropriate  member  of  the  OT&E 
team  to  witness  any  verification  event.  The 
team  member  witnessing  the  verification 
event  is  responsible  for  documenting  the 
verification  results  in  accordance  with  the 
V&V  Observation  Report  template  (notice 
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the  similarity  to  the  DT  Observation 
Report)  as  shown  in  this  chapter.  The 
ACA  will  reference  this  documentation, 
and  may  need  additional  information 
from  the  OT&E  team  member  or  suitable 
government  representative  that  witnessed 
the  event,  when  checking  the  V&V  report 
for  accuracy. 

Validation  Process 

Basics  of  Validation 

Reference  (DODI  5000.61)  defines 
validation  as,  “The  process  of  determining 
the  degree  to  which  a  model  or  simulation 
and  its  associated  data  are  an  accurate 
representation  of  the  real  world  from  the 
perspective  of  the  intended  uses  of  the 
model.”  Validation  is  accomplished  by 
comparing  the  output  of  an  M&S,  with 
respect  to  its  intended  uses,  to  real  world 
known  or  expected  behavior  of  the  subject 
it  represents.  In  order  to  be  valid,  the  M&S 
output  must  replicate  the  real  world  subject 
being  modeled  within  the  established 
degree  of  fidelity.  If  an  M&S  does  not 
produce  valid  representations  of  the  real 
world  system  or  processes  in  question, 
conclusions  based  on  using  the  M&S  will 
be  erroneous  resulting  poor  decisions  or 
actions.  Therefore  it  is  essential  to  establish 
the  validity  of  the  M&S  prior  to  using  it  to 
support  any  decisions  or  actions. 

M&S  are  used  to  support  OT&E  when 
using  the  actual  systems  or  processes 
being  modeled  to  gather  sufficient  data  are 
impossible,  unsafe,  or  impractical.  Since  an 
M&S  represents  an  approximation  of  the 
real  world,  it  will  always  have  limitations. 

A  given  M&S  will  never  be  absolutely 
valid.  For  this  reason  the  SIUs  in  support 
of  OT&E  are  identified  early  so  the  V&V 
effort  remains  focused  on  the  right  M&S 
uses.  M&S  validation  activities  should  be 
accomplished  with  an  eye  toward  the  SIUs. 
This  is  not  meant  to  limit  M&S  validation 
to  just  an  examination  of  the  SIUs  with 
respect  to  their  associated  accreditation 
criteria  (although  this  is  a  key  part  of 


validation).  However,  all  validation  activity 
should  be  related  to  the  SIUs  to  avoid 
needlessly  expending  scarce  resources  by 
attempting  to  make  an  M&S  more  capable 
than  it  needs  to  be. 

All  aspects  of  the  M&S  require  validation 
to  include  the  model  itself,  the  data  used 
by  the  model,  any  look-up  tables  used, 
any  extrapolation  techniques  used,  any 
methodologies  used  that  are  external  to  the 
M&S,  and  any  required  interfaces  between 
the  M&S  and  another  M&S,  system, 
or  entity.  The  M&S  may  be  validated  in 
pieces,  but  it  shall  also  be  validated  in  its 
final  configuration,  using  the  applicable 
input  data,  as  it  will  be  run  during 
support  to  OT&E.  If  an  M&S  consists 
of  a  federation  of  models,  the  federated 
M&S  shall  be  validated  as  it  is  intended 
to  be  run.  Even  if  all  the  components  of 
the  federation  have  been  independently 
validated,  the  federation  of  models  shall  be 
validated  while  functioning  as  the  intended 
federation.  The  OT&E  team  member 
(likely  the  ACA)  witnessing  the  validation 
event  is  responsible  for  documenting  the 
validation  results  in  accordance  with  the 
V&V  Observation  Report  template  (notice 
the  similarity  to  the  DT  Observation 
Report)  as  shown  in  this  chapter. 

The  following  section  describes  some 
common  validation  techniques.  For  more 
information  on  these  and  other  techniques, 
see  reference  (DOD  2001).  The  more 
validation  techniques  used  to  successfully 
validate  an  M&S  functionality,  the  more 
confident  the  MCOTEA  accreditation 
agent  and  authority  will  be  that  the 
M&S  is  a  credible  representation  of  that 
functionality. 

Common  Validation  Techniques 

Using  Data  From  the  Modeled  System 
or  Environment 

If  the  M&S  represents  the  operation 
of  an  existing  system,  the  best  means  of 
validation  is  to  compare  M&S  results 
to  the  behavior  of  the  actual  system 
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under  as  close  to  identical  conditions  as 
possible.  This  can  be  problematic,  in  that 
the  M&S  conditions  can  be  precisely 
specified,  while  the  operating  conditions 
of  the  actual  system,  although  as  tightly 
controlled  as  possible,  may  result  in  sources 
of  comparison  error.  The  comparison 
errors  introduced  by  real  world  operations 
must  be  accounted  for  in  the  validation 
criteria.  One  way  to  define  validity  is  if  the 
M&S  results  fall  within  a  specified  error 
interval,  say  ±10%,  at  a  desired  confidence 
level,  say  80%.  Note  that  the  error  interval 
defines  whether  a  particular  M&S  result 
compared  to  the  real  world  is  valid,  while 
the  confidence  level  defines  the  number 
of  trials  required.  The  error  interval  and 
confidence  level  together  set  the  validation 
criteria  for  each  validation  check. 

However,  validation  of  the  M&S  shall 
be  accomplished  regardless  of  whether 
or  not  the  corresponding  real  world 
system  currently  exists  or  the  real  world 
environment  is  available  for  comparison.  If 
a  system  corresponding  to  the  M&S  does 
not  currently  exist,  or  the  environment 
is  not  available  for  comparison,  there  are 
other  validation  options. 

Using  Data  From  Related,  Existing 
Systems  or  Environments 

Lacking  an  existing  system  or  suitable 
environment  from  which  to  gather 
validation  data,  data  from  a  related,  existing 
system  or  environment  can  be  used  to  help 
validate  the  M&S.  The  technique  would 
be  to  construct  a  preliminary  M&S  of  the 
existing  system  or  related  environment  and 
perform  a  validation  of  this  preliminary 
M&S.  Once  the  preliminary  M&S  is 
suitably  validated,  it  is  modified  to  create  the 
desired  M&S  that  represents  the  proposed 
system  or  environment.  The  fewer  the 
modifications  needed  to  the  preliminary 
(validated)  M&S,  the  higher  the  confidence 
in  the  desired  M&S.  Greater  confidence 
in  an  M&S  constructed  in  this  way  can  be 
obtained  by  employing  some  of  the  other 
techniques  in  this  section. 


Using  SMEs 

Whether  or  not  there  is  an  existing  system 
or  suitable  environment  corresponding 
to  the  M&S,  SMEs  can  be  helpful  in 
building  confidence  that  an  M&S  is 
valid.  SMEs  view  the  M&S  output  under 
various  conditions  for  reasonableness, 
based  on  their  experience.  The  selection 
of  SMEs  is  important.  SMEs  should  be 
experts  in  the  warfare  area  or  technical 
area  corresponding  to  or  using  the  system 
being  modeled  by  the  M&S  or  experts 
in  systems  similar  to  the  system  being 
modeled.  The  SMEs  should  also  have 
an  understanding  of  the  OT&E  SIUs 
designated  for  the  M&S.  If  SMEs  are  used 
for  validation,  more  than  one  should  be 
used  and  they  must  come  to  a  consensus 
before  the  validation  is  useful.  If  the  SMEs 
think  the  M&S  results  are  reasonable,  that 
strengthens  the  case  for  validation. 

A  useful  exercise,  called  a  Turing  Test,  is  to 
show  the  SMEs  data  from  the  real  world  and 
corresponding  data  from  the  M&S  without 
knowing  the  sources  of  the  data  sets.  If  the 
SMEs  can  accurately  discriminate  between 
the  two  data  sets,  the  reasons  they  cite  can 
be  useful  in  correcting  errors  in  the  M&S.  If 
the  SMEs  cannot  agree  on  the  sources  of  the 
data  sets,  that  is  another  argument  in  favor  of 
M&S  validation  for  the  uses  implied  by  the 
data  sets. 

Using  Another  Model 

Once  MCOTEA  has  accredited  a  model 
for  a  specific  use,  the  results  of  that 
specific  version  of  the  model  are  trusted 
by  MCOTEA  for  that  specific  usage. 
Therefore,  the  previously  accredited 
model  may  be  used  to  validate  the  results 
of  another  model,  as  long  as  it  is  for  the 
previously  accredited  usage. 

This  validation  technique  should  be  used 
with  caution  for  the  following  reasons. 
Typically  a  model  is  considered  accurate  if 
its  results  fall  within  the  desired  accuracy  of 
the  real-world  results.  If  this  accuracy  were 
say  ±10  percent  of  the  real-world  value,  the 
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first  model  (model  A)  could  be  as  much 
as  9.999  percent  off  and  still  pass.  If  this 
model  were  then  used  to  validate  another 
model  (model  B),  and  model  B  was  also  off 
of  the  model  A  results  by  9.9999  percent 
in  the  same  direction,  model  B  would  be 
close  to  20  percent  off  the  real-world  value. 
But  since  it  was  less  than  10  percent  off  the 
model  A  results,  model  B  would  still  pass 
using  this  validation  method. 

Another  reason  this  technique  should  be 
used  with  caution  is  the  situation  where 
model  B  is  based  on  model  A.  This  often 
happens  when  there  is  a  desire  to  improve 
model  A.  If  there  were  an  undiscovered 
systematic  error  in  model  A,  and  model  B 
were  based  on  model  A,  this  undiscovered 
error  would  probably  be  conveyed  to  model 
B,  the  daughter  of  model  A.  If  model  A 
were  used  to  then  validate  model  B,  the 
error  would  never  be  discovered,  since  model 
B  would  reproduce  the  same  erroneous 
results  as  the  original  model  A.  Under  these 
circumstances,  model  A  could  only  be  used 
to  verify  that  there  were  no  new  errors 
introduced  during  the  coding  of  model  B. 

Therefore,  using  a  previously  accredited 
model  (model  A)  to  validate  second  model 
(model  B)  can  only  be  done  under  the 
following  circumstances: 

♦  MCOTEA  has  previously  accredited  model 
A  for  the  SIUs  under  consideration 

♦  Model  B  was  constructed  independently 
from  model  A,  that  is  model  B  is  not  a 
daughter  of  model  A 

♦  The  usage  being  validated  for  model  B  is 
identical  to  that  previously  accredited  for 
model  A 

♦  If  a  future  model  (model  C)  uses  this 
validation  technique,  only  the  originally 
accredited  model  may  be  used  as  the 
validation  tool,  that  is,  model  A  can  be 
used  to  validate  model  C  for  a  previously 
accredited  usage,  but  model  B  (validated 
using  model  A  results)  cannot.  An  exception 
to  this  rule  is  if  model  B  was  derived  from 
model  A  and  model  C  is  derived  from 
model  B.  Model  A  cannot  be  used  to 


validate  model  C,  its  granddaughter. 

Using  Predictions 

A  prediction  is  obtained  by  running 
an  M&S  under  conditions  that  will  be 
experienced  in  the  future.  Predictions  are 
useful  since  there  is  no  way  to  consciously  or 
unconsciously  “back  in”  the  model  results. 
The  M&S  is  run,  predicted  values  and  data 
are  recorded,  and  the  M&S  results  are  then 
compared  to  the  real  world  results  at  some 
future  time  when  the  predicted  conditions 
are  experienced.  Predictions  accurate  to 
the  required  level  of  fidelity  support  M&S 
validation.  Predictions  can  also  be  used  to 
discover  errors  in  the  M&S  or  to  update 
parameter  values  when  the  M&S  results 
disagree  with  the  real-world  results. 

Sensitivity  Analyses 

Sensitivity  analysis  determines  factors 
having  the  greatest  impact  on  M&S 
results  and  that  should  be  modeled  most 
carefully.  Clearly  sensitivity  analyses  can  be 
used  to  locate  coding  errors  and  might  be 
considered  part  of  the  verification  process. 
However,  unexpected  behavior  during 
sensitivity  analysis  might  indicate  invalid 
behavior  as  well.  If  small  changes  in  a 
value  correspond  to  large  changes  in  M&S 
output,  sensitivity  analysis  will  also  reveal 
those  values  that  need  to  be  specified  with 
the  most  accuracy. 

Candidates  for  sensitivity  analysis  in  the 
M&S  are: 

♦  Parameter  values 

♦  Probability  distribution  selection 

♦  Assumptions 

These  things  should  be  chosen  in  a  way 
that  most  closely  represents  reality. 
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Typically,  MCOTEA  is  interested  in 
using  the  data  that  is  output  from  an 
M&S  to  support  some  aspect  of  OT&E. 
However,  many  models  require  certain 
parameters  be  set  or  certain  data  be  input 
in  order  for  them  to  produce  the  needed 
output.  These  input  data  can  be  fed 
into  the  M&S  by  an  operator  in  order 
to  fill  required  data  fields  prior  to  each 
run  of  an  M&S,  or  these  data  might  be 
“hardwired”  as  fixed  parameters  within 
the  model  itself  The  accuracy  of  M&S 
output  is  just  as  dependent  on  the  input 
data  as  it  is  on  the  accuracy  of  the  M&S 
itself  Therefore,  in  addition  to  any  M&S 
to  be  used  by  MCOTEA  in  support  of 
OT&E,  any  data  used  as  input  to  the 
M&S,  or  as  fixed  parameters  within  an 
M&S  must  be  verified  and  validated. 

Ref  (MIL-STD-3022)  defines  data 
V&V  as,  “The  process  of  verifying  the 
internal  consistency  and  correctness  of 
data  and  validating  that  it  represents 
real-world  entities  appropriate  for  its 
intended  purpose  or  an  expected  range  of 
purposes.”The  types  of  data  that  require 
V&V  are  data  used: 

♦  To  verify  M&S  requirements 

♦  To  verify  M&S  accuracy 

♦  To  build  the  conceptual  model 

♦  To  validate  the  M&S 

♦  To  perform  experiments  in  support  of 
OT&E  or  M&S  V&V 

♦  To  run  combat  support  decision  aids 

♦  As  input  to  any  M&S  supporting 
OT&E 

Even  if  the  data  are  consistent  and 
accurate,  the  data  set  may  not  be  suitable 
for  a  given  application.  The  data  might 
be  incompatible  with  the  application,  it 
might  generated  based  on  assumptions 
that  are  not  compatible  with  the 
M&S  assumptions,  or  it  might  not 
have  been  generated  at  an  appropriate 
level  of  fidelity.  Given  this,  any  data 
requiring  V&V  must  be  accompanied  by 


information  concerning  the  original  reason 
the  data  was  generated,  how  the  data  was 
generated  (the  more  detail  the  better),  and 
any  assumptions  made  in  generating  the 
data.  This  will  give  the  ACA  information 
pertaining  to  the  quality  of  the  data  and  if 
the  data  is  appropriate  for  the  intended  use. 

The  age  of  a  data  set  is  irrelevant.  As  long 
as  the  data  can  be  V&V’d  in  accordance 
with  this  chapter,  it  may  be  used  to  support 
MCOTEA  OT&E. 

Data  Verification 

The  verification  of  data  focuses  on  its 
accuracy.  The  idea  is  to  ensure  the  data  has 
been  accurately  translated,  is  complete, 
is  credible,  is  interpreted  correctly  when 
used  by  the  M&S,  and  supports  the  input 
requirements  of  the  M&S.  Data  can  be 
verified  by  inspection  using  a  process 
much  like  proof-reading;  it  helps  ensure 
the  data  isn’t  inadvertently  changed  when 
transcribing  it  from  its  point  of  generation 
to  the  M&S  input.  A  SME  is  useful  in  data 
verification,  since  a  SME  can  often  identify 
data  that  appear  unreasonable  under  a  given 
set  of  conditions.  A  SME  can  help  decide  if 
the  data  comes  from  a  credible  source  and 
that  the  data  has  been  interpreted  correctly 
when  translated  into  M&S  parameters. 

Another  aspect  of  data  verification  is 
ensuring  it  comes  in  the  expected  form  and 
is  properly  prepared  for  use  in  the  M&S. 
For  example,  phone  numbers  in  the  United 
States  come  in  the  form  xxx-xxx-xxxx.  A 
data  entry  (phone  number)  not  conforming 
to  this  form  may  be  erroneous.  A  more 
sophisticated  verification  check  on  this  data 
might  involve  ensuring  the  first  3  digits 
represent  a  valid  area  code  within  the  U.S. 

From  the  MCOTEA  perspective,  the 
ACA  must  ensure  that  data  verification 
procedures  for  the  M&S  are  in  place,  are 
being  executed,  and  all  input  data  are 
verified  before  M&S  execution  in  support 
of  OT&E.  All  data  verification  activities 
and  processes  shall  be  documented  in 
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the  V&V  Report  and  the  Accreditation 
Report. 

Data  Validation 

Data  is  validated  to  ensure  it  accurately 
and  adequately  represents  the  real  world 
to  be  simulated.  Data  is  validated  by 
comparing  it  to  a  set  of  acceptance  criteria. 
The  acceptance  criteria  are  crafted  in  a  way 
that  ensures  the  data  set  will  be  acceptable 
for  its  intended  use,  therefore,  the  ACA 
must  approve  all  data  acceptance  criteria 
applicable  to  MCOTEA  SIUs. 

One  way  to  validate  a  data  set  is  to  compare 
it  to  real  world  data  and  establish  the 
degree  to  which  the  two  data  sets  must 
match.  In  some  applications,  the  data  input 
to  an  M&S  comes  directly  from  the  real 
world.  For  example,  if  an  M&S  models  the 
performance  of  a  given  radar  system,  and 
the  M&S  uses  the  antenna  pattern  obtained 
from  the  actual  radar  it  is  intended  to  model, 
the  antenna  pattern  already  represents 
validated  data  because  it  comes  directly  from 
the  real  world  system  being  modeled. 

Data  can  also  be  validated  by  comparing 
it  to  an  analogous  real  world  system/ 
situation,  again  within  the  constraints 
of  the  approved  acceptance  criteria.  In 
the  example  above,  the  antenna  pattern 
to  be  used  by  the  first  M&S  might  be 
generated  by  another  M&S.  The  computer¬ 
generated  antenna  pattern  can  be  validated 
by  comparing  it  to  the  actual  antenna 
pattern.  If  the  computer-generated 
pattern  compares  favorably  to  the  real 
antenna  pattern  within  previously  agreed 
upon  acceptable  limits  (standards),  the 
computer-generated  pattern  is  considered 
validated  for  use  by  the  first  M&S. 

Data  validation  can  be  assisted  by  SME 
inspection.  SMEs  view  the  data  under 
various  conditions  for  reasonableness,  based 
on  their  experience.  The  selection  of  SMEs 
is  important.  SMEs  should  be  experts  in  the 
warfare  area  or  technical  area  corresponding 
to  the  system  being  modeled  by  the  M&S 
or  in  systems  similar  to  the  system  being 


modeled.  The  SMEs  should  also  have 
an  understanding  of  the  OT&E  SIUs 
corresponding  to  the  data  under  examination. 
If  SMEs  are  used  for  data  validation,  more 
than  one  should  be  used  and  they  must  come 
to  a  consensus  before  the  data  validation  is 
useful.  If  the  SMEs  think  the  data  being 
validated  are  reasonable,  that  strengthens  the 
case  for  data  validation. 

In  general,  in  order  to  be  validated,  any 
data  used  as  input  to  an  M&S  intended 
for  use  by  MCOTEA  must  either  be 
the  relevant  real  world  data  itself,  or 
it  must  compare  favorably  within  pre¬ 
defined,  acceptable  limits  to  the  relevant 
real  world  data.  The  ACA  shall  ensure 
that  all  input  data  intended  to  support 
MCOTEA  SIUs  is  validated  against  the 
approved  acceptance  criteria.  The  data 
acceptance  criteria  shall  be  defined  in  the 
Accreditation  Plan,  and  the  validation  shall 
be  documented  in  the  V&V  Report  and 
the  Accreditation  Report. 

Use  of  Surrogate  Data 

It  is  always  preferable  to  use  the  data 
explicitly  required  by  the  M&S;  however, 
occasionally  the  data  required  as  input 
to  an  M&S  may  not  exist.  In  such  cases 
similar  data  may  exist  and  can  be  used 
to  approximate  the  desired  M&S  input 
data.  For  example,  controlled  data  on  how 
human  skin  reacts  to  heat  (human  burn 
data)  might  be  hard  to  find  or  might  not 
exist.  However,  controlled  experiments 
dealing  with  how  animal  skin  reacts  to  heat 
does  exist.  In  this  case,  it  will  be  necessary 
to  run  the  M&S  based  largely  on  the 
animal  data,  then  extrapolate  the  M&S 
output  to  effects  on  humans. 

The  extrapolation  of  the  surrogate  data 
is  part  of  the  model,  so  it  (the  surrogate 
data  and  the  extrapolation  technique) 
must  be  V&V’d.  Evidence  supporting  the 
verification  and  validation  of  the  surrogate 
data  and  the  extrapolation  technique  shall 
be  included  in  the  Accreditation  Report. 
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Acquisition  Category 

Categories  established  to  facilitate  decentralized  decision-making  and  execution  and 
compliance  with  statutorily  imposed  requirements.  Acquisition  Categories  (ACAT) 
determine  the  level  of  review,  decision  authority,  and  applicable  procedures  (CJCSI  2005, 
DAU  2005).  (For  ACAT  categories,  see  chapter  2  of  this  manual  or  SECNAVINST 
2008.) 

Analytic  Model 

A  model  that  focuses  on  the  COIs,  composed  of  terms  reflecting  Performance,  Suitability, 
and  Survivability,  meaning  that  an  analytic  model  for  the  COIs  should  incorporate  all 
three  of  these  dimensions.  Incorporating  Suitability  and  Survivability  parameters  into  the 
analytic  model  is  critical  to  determining  their  relative  impact  on  Effectiveness. 

Attribute 

A  quantitative  or  qualitative  characteristic  of  an  element  or  its  actions  (CJCSI  2005). 

Availability 

A  measure  of  the  degree  to  which  an  item  is  in  an  operable  state  and  can  be  committed 
at  the  start  of  a  mission  when  the  mission  is  called  for  at  an  unknown  (random)  point  in 
time  (DAU  2005). 

Capability 

The  ability  to  achieve  a  desired  effect  under  specified  standards  and  conditions  through 
combinations  of  means  and  ways  to  perform  a  set  of  tasks.  Capability  is  defined 
by  an  operational  user  and  expressed  in  broad  operational  terms  in  the  format  of  a 
Joint  or  Initial  Capabilities  Document  or  a  Joint  doctrine,  organization,  training, 
materiel,  leadership  and  education,  personnel,  and  facilities  (DOTMLPF)  change 
recommendation.  In  the  case  of  materiel  proposals,  the  definition  will  progressively 
evolve  to  DOTMLPF  performance  attributes  identified  in  the  Capability  Development 
Document  and  the  Capability  Production  Document  (CJCSI  2005). 

Capability  Development  Document 

A  programmatic  document  created  by  DC,  CD&I  that  captures  the  information 
necessary  to  develop  a  proposed  program,  normally  using  an  evolutionary  acquisition 
strategy.  The  Capability  Development  Document  (CDD)  outlines  an  affordable 
increment  of  militarily  useful,  logistically  supportable,  and  technically  mature  capability. 
The  CDD  supports  a  Milestone  B  decision  review  (DAU  2005).  MCOTEA  derives 
Issues,  Attributes,  and  Measures  from  the  capabilities  in  the  CDD.  CJCSM  3170.01B 
contains  the  CDD  format. 

Capability  Production  Document 

A  programmatic  document  created  by  DC,  CD&I  that  addresses  the  production 
elements  specific  to  a  single  increment  of  an  acquisition  program.  The  Capability 
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Production  Document  (CPD)  must  be  validated  and  approved  before  a  Milestone 
C  decision  review.  The  refinement  of  performance  Attributes  and  Key  Performance 
Parameters  is  the  most  significant  difference  between  the  Capability  Development 
Document  and  CPD  (DOD  2008  and  CJCSI  2005).  MCOTEA  derives  Issues, 
Attributes,  and  Measures  from  the  capabilities  in  the  CPD.  CJCSM  3170. 01B  contains 
the  CDD  format. 

Chargeability 

The  characterization  of  a  test  incident  or  failure  by  the  reason,  component,  or  process 
to  which  the  event  can  be  attributed.  Such  a  characterization  is  agreed  upon  during  the 
Failure  Definition  /Scoring  Criteria  Conference  and  affects  the  calculation  of  Reliability, 
Availability,  and  Maintainability  (RAM)  metrics. 

Collectively  Exhaustive 

If  the  evaluation  covers  every  mission  required  of  the  system  as  well  as  all  relevant  aspects 
of  Suitability  and  Survivability,  then  the  evaluation  will  be  collectively  exhaustive. 

Command  and  Control 

The  exercise  of  authority  and  direction  by  a  properly  designated  commander  over 
assigned  and  attached  forces  in  the  accomplishment  of  the  mission.  Command  and 
control  functions  are  performed  through  an  arrangement  of  personnel,  equipment, 
communications,  facilities,  and  procedures  employed  by  a  commander  in  planning, 
directing,  coordinating,  and  controlling  forces  and  operations  in  the  accomplishment  of 
the  mission.  Also  called  C2  (DOD  2011). 

Command  and  Control  System 

The  facilities,  equipment,  communications,  procedures,  and  personnel  essential  to  a 
commander  for  planning,  directing,  and  controlling  operations  of  assigned  and  attached 
forces  pursuant  to  the  missions  assigned  (DOD  2011). 

Concept  of  Employment 

A  statement  that  portrays  how  a  user  may  employ  a  system  under  development  while 
conducting  a  mission.  The  Concept  of  Employment  (COE)  typically  provides  a  system 
description  and  addresses  operational  employment,  platform  applications,  and  associated 
command  and  control  considerations  for  the  system.  MCOTEA  uses  the  COE  to 
develop  the  test  concept  as  part  of  the  Program  Definition  phase  of  the  Detailed  Test 
Plan  development.  In  general,  the  COE  may  be  found  in  the  Capabilities  Production 
Document,  but  MCOTEA  should  coordinate  with  the  DC,  CD&I  action  officer  for 
direction  in  defining  the  COE. 

Concept  of  Operations 

Verbal  or  graphic  statement,  in  broad  outline,  of  a  commander’s  assumptions  or  intent  in 
regard  to  an  operation  or  series  of  operations.  The  concept  of  operations  (CONOPS)  is 
frequently  embodied  in  campaign  plans  and  operation  plans;  in  the  latter  case,  particularly 
when  the  plans  cover  a  series  of  connected  operations  to  be  carried  out  simultaneously 
or  in  succession.  The  concept  is  designed  to  give  an  overall  picture  of  the  operation  and 
is  included  primarily  for  additional  clarity  of  purpose.  Also  called  commander’s  concept 
(CJCSI  2002). 
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Consolidated  Review  Board 

Review  Board  consisting  of  representatives  from  MCOTEA  to  discuss  and  concur  on 
Plans  and  Reports.  The  Consolidated  Review  Board  (CRB)  ensures  that  the  planned 
effort  depicted  in  the  document  is  complete,  adequate,  defensible,  consistent  with  the 
TEMP,  and  able  to  provide  the  data  required  to  resolve  all  Issues  not  precluded  by  test 
limitations.  The  Report  CRB  provides  final  guidance  to  the  test  team  and  approves  the 
evaluation  and/or  assessment. 

Critical  Operational  Issue 

Critical  Operational  Issues  (COI)  are  key  Operational  Effectiveness  or  Suitability  Issues 
that  must  be  examined  in  OT&E  to  determine  the  system’s  capability  to  perform  its 
mission.  COIs  must  be  relevant  to  the  required  capabilities  and  of  key  importance  to  the 
system  being  OE/OS/OSur  and  represent  a  significant  risk  if  not  satisfactorily  resolved. 

A  COI  is  normally  phrased  as  a  question  that  must  be  answered  in  the  affirmative  to 
properly  evaluate  operational  effectiveness  (e.g.,“Will  the  system  detect  threats  in  a 
combat  environment  at  adequate  range  to  allow  successful  engagement?”)  (DAU  2005). 

A  COI  may  be  decomposed  into  a  set  of  performance,  suitability,  and  survivability 
Measures. 

Data  Collection  Verification  and  Validation 

An  exercise  that  tests  the  data  collection  methodology.  Data  Collection  V&V  (DCV&V) 
ensures  that  data  collection  equipment  functions  properly  and  reliably  and  that  data 
collection  forms  adequately  capture  required  data.  Data  Collection  V&V  verifies  the 
adequacy  of  the  data  collection  plan  designed  for  the  system  under  test  and  validates  the 
accuracy  and  completeness  of  the  resulting  data  reports  in  resolving  the  Detailed  Test 
Plan  Measures. 

Developmental  Test  and  Evaluation 

Testing  done  by  MCSC/PEO  LS  to  verify  the  status  of  technical  progress,  to  verify 
that  design  risks  are  minimized,  to  substantiate  achievement  of  contract  technical 
performance,  and  to  certify  readiness  for  operational  testing.  Developmental  tests 
generally  require  instrumentation  and  measurements  and  are  accomplished  by  engineers, 
technicians,  or  Marine  operator-maintainer  test  personnel  in  a  controlled  environment  to 
facilitate  failure  analysis.  Any  testing  used  to  assist  in  the  development  and  maturation  of 
products,  product  elements,  or  manufacturing  or  support  processes  (DAU  2005). 

Evaluation  Framework 

Identifies  the  Evaluation  Questions  (COIs  and  Issues)  that  must  be  answered  along  with 
their  Standards  and  Measures.  The  Evaluation  Framework  also  provides  the  traceability  of 
Attributes  back  to  the  capabilities  documents. 

Failure  Definition/Scoring  Criteria  Charter 

An  agreement  between  MCOTEA;  DC,  CD&I;  and  MCSC/PEO  LS,  signed  before 
test  execution,  for  anticipating  the  characterization  of  test  incidents  used  to  evaluate 
Reliability,  Availability,  and  Maintainability  Measures.  See  also  chargeability. 
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Failure  Definition/Scoring  Criteria  Scoring  Conference 

Posttest  conference  that  allows  MCOTEA;  DC,  CD&I;  and  MCSC/PEO  LS  to 
adjudicate  the  scoring  of  test  incidents.  Outcomes  affect  Reliability,  Availability,  and 
Maintainability,  as  well  as  other  performance  Measures.  See  also  chargeability. 

Feasibility  of  Support  Message 

Naval  message  outlining  requirements  for  Marine  operating  forces  personnel  and 
facilities;  generated  by  OTPO/S-3. 

Follow-On  Operational  Test  and  Evaluation 

The  Test  and  Evaluation  that  may  be  necessary  after  the  Full-Rate  Production  Decision 
Review  to  refine  the  estimates  made  during  IOT&E,  to  evaluate  changes,  and  to 
reevaluate  the  system  to  ensure  that  it  continues  to  meet  operational  needs  and  retains  its 
effectiveness  in  a  new  environment  or  against  a  new  threat  (DAU  2005). 

Full-Rate  Production 

Economical  production  quantities  following  stabilization  of  the  system  design  and 
validation  of  the  production  process  (DAU  2005). 

Fully  Mission  Capable 

The  system,  in  the  mission  context,  has  achieved  at  least  the  equivalent  of  threshold 
performance  or  better  for  the  desired  effect  or  outcome. 

Functional  Needs  Analysis 

Assesses  the  ability  of  the  current  and  programmed  warfighting  systems  to  deliver  the 
capabilities  that  the  Functional  Area  Analysis  identified  under  the  full  range  of  operating 
conditions  and  the  designated  Measures.  The  Functional  Needs  Analysis  (FNA)  produces 
a  list  of  capability  gaps  that  require  solutions  and  indicates  the  time  frame  in  which 
those  solutions  are  needed.  It  may  also  identify  redundancies  in  capabilities  that  reflect 
inefficiencies  (CJCSI  2005). 

“Ilities” 

The  operational  and  support  requirements  that  a  program  must  address  (e.g.,  availability, 
maintainability,  vulnerability,  reliability,  logistics  supportability,  etc.)  (DAU  2005). 

Information  Assurance 

Information  operations  that  protect  and  defend  information  and  information  systems  by 
ensuring  their  availability,  integrity,  authentication,  confidentiality,  and  non-repudiation. 
This  includes  providing  for  restoration  of  information  systems  by  incorporating 
protection,  detection,  and  reaction  capabilities  (CJCSI  2005). 

Information  Support  Plan 

The  identification  and  documentation  of  information  needs,  infrastructure  support, 

IT  and  NSS  interface  requirements,  and  dependencies  focusing  on  net-centric, 
interoperability,  supportability,  and  sufficiency  concerns  (DODI  2004). 
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Initial  Capabilities  Document 

A  programmatic  document  created  by  DC,  CD&I  that  addresses  the  need  to  resolve  a 
specific  capability  gap,  or  set  of  capability  gaps,  identified  through  the  JCIDS  analysis 
process.  The  Initial  Capabilities  Document  (ICD)  defines  the  gap  in  terms  of  the 
functional  area;  the  relevant  range  of  military  operations;  desired  effects;  time;  Doctrine, 
Organization,  Training,  Materiel,  Leadership  and  Education,  Personnel,  and  Facilities 
(DOTMLPF);  and  policy  implications  and  constraints.  The  outcome  of  an  ICD  could 
be  one  or  more  DOTMLPF  Change  Recommendations  or  Capability  Development 
Documents.  ICDs  should  be  non-system  specific  and  non-Service,  agency  or  activity- 
specific  to  ensure  capabilities  are  being  developed  in  consideration  of  the  joint  context 
(CJCSI  2005).  MCOTEA  derives  COIs,  Issues,  and  Measures  from  the  capabilities  in 
the  ICD. 

Initial  Operational  Test  and  Evaluation 

Operational  test  and  evaluation  conducted  on  production  or  production-representative 
articles  to  determine  whether  systems  are  operationally  effective  and  suitable,  and  which 
supports  the  decision  to  proceed  beyond  Low- Rate  Initial  Production  (DAU  2005). 

Integrated  Product  Team 

Team  composed  of  representatives  from  appropriate  functional  disciplines  working 
together  to  build  successful  programs,  identify  and  resolve  issues,  and  make  sound  and 
timely  recommendations  to  facilitate  decision-making.  Three  types  of  Integrated  Product 
Teams  (IPT)  exist:  Overarching  IPTs,  which  focus  on  strategic  guidance,  program 
assessment,  and  issue  resolution;  Working-level  IPTs,  which  identify  and  resolve  program 
issues,  determine  program  status,  and  seek  opportunities  for  acquisition  reform;  and 
Program-level  IPTs,  which  focus  on  program  execution  and  may  include  representatives 
from  both  government  and  industry  after  contract  award  (DAU  2005). 

Integrated  Testing 

The  collaborative  planning  and  collaborative  execution  of  test  phases  and  events  to 
provide  shared  data  in  support  of  independent  analysis,  evaluation,  and  reporting  by 
all  stakeholders,  particularly  the  developmental  (both  contractor  and  government)  and 
operational  test  and  evaluation  communities. 

Intermediate  Assessments 

Intermediate  Assessments  pertain  to  assessment  of  Developmental  Testing  results. 
Intermediate  Assessments  may  be  performed  using  one  or  more  DT  Reports.  Intermediate 
Assessment  is  also  performed  when  MCOTEA  plans  and  executes  a  DT  event. 

Intermediate  Assessment  Report 

After  one  or  more  DT  Observations,  MCOTEA  writes  an  Intermediate  Assessment 
Report  upon  reciept  of  DT  data  or  reports.  The  PM  and  MDA  use  these  reports  to  gauge  a 
program’s  progress  toward  IOT  and  to  become  aware  of  any  risks  to  program  success. 

Issues 

Any  aspect  of  the  system’s  capability,  either  operational,  technical  or  other,  that  must  be 
questioned  before  the  system’s  overall  military  utility  can  be  known.  Operational  Issues 
are  Issues  that  must  be  evaluatied  considering  the  Warfighter  and  the  machine  as  an 
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entity  to  estimate  the  OE/OS  of  the  system  in  its  complete  user  environment  (DAU 
2005). 

Joint  Acquisition  Program 

A  directed  joint  effort  for  the  development  and  procurement  of  systems,  subsystems, 
equipment,  software,  or  munitions  as  well  as  supporting  equipment  or  systems,  with  the 
goal  of  providing  a  new  or  improved  capability  for  a  validated  joint  need  (DAU  2005). 

Joint  Capabilities  Document 

A  programmatic  document  that  identifies  a  set  of  capabilities  that  support  a  defined 
mission  area  utilizing  associated  Family  of  Joint  Future  Concepts,  Concept  of  Operations, 
or  Unified  Command  Plan-assigned  missions.  The  Joint  Capabilities  Document  ( JCD) 
will  be  updated  as  changes  are  made  to  the  supported  Family  of  Joint  Future  Concepts, 
CONOPS,  or  assigned  missions.  MCOTEA  should  correspond  with  the  DC,  CD&I 
action  officer  for  other-Service  documents  related  to  the  system  (CJCSI  2005). 

Joint  Capabilities  Integration  and  Development  System 

Joint  Capabilities  Integration  and  Development  System  (JCIDS)  supports  the  Chairman 
of  the  Joint  Chiefs  of  Staff  and  the  Joint  Requirements  Oversight  Council  in  identifying, 
assessing,  and  prioritizing  joint  military  capability  needs  as  required  by  law.  The 
capabilities  are  identified  by  analyzing  what  is  required  across  all  functional  areas  to 
accomplish  the  mission  (DAU  2005).  See  also  CJCSI  3170. 01E. 

Joint  Operations  Concepts 

The  Joint  Operations  concepts  (JOpsC)  is  the  overarching  concept  that  guides  the 
development  of  future  joint  force  capabilities.  It  broadly  describes  how  the  joint  force 
is  expected  to  operate  10-20  years  in  the  future  in  all  domains  across  the  range  of 
military  operations  within  a  multilateral  environment  in  collaboration  with  interagency 
and  multinational  partners.  The  JOpsC  describes  the  proposed  end  states  derived  from 
strategy  as  military  problems  and  the  key  characteristics  of  the  future  joint  force  (CJCSI 
2005). 

Joint  Program 

Any  defense  acquisition  system,  subsystem,  component,  or  technology  program  that 
involves  formal  management  or  funding  by  more  than  one  DOD  Component  during  any 
phase  of  a  system’s  life  cycle  (DAU  2005). 

Key  Performance  Parameter 

Those  Attributes  or  characteristics  of  a  system  that  are  considered  critical  or  essential 
to  the  development  of  an  effective  military  capability  and  those  Attributes  that  make 
a  significant  contribution  to  the  key  characteristics  as  defined  in  the  Joint  Operations 
Concept,  i.e.,  communications,  interoperability,  etc.  (CJCSI  2005).  DC,  CD&I  designates 
which  capabilities  are  Key  Performance  Parameters  (KPP)  in  programmatic  documents. 

Lethality  Testing 

Fethality  is  the  weapons  system’s  ability  to  cause  the  loss  of,  or  the  degradation  in,  the 
target  system’s  ability  to  complete  its  designated  mission. 


G-7 


Glossary 


Limited  User  Evaluations 

A  limited  evaluation  of  a  system  operated  by  the  intended  user  of  a  system  in  an 
operational  setting  in  accordance  with  the  Concept  of  Employment  and/or  the 
Operational  Mode  Summary/Mission  Profile.  Cannot  be  used  to  determine  OE/OS/ 
OSur. 

Live  Fire  Test  and  Evaluation 

A  test  process  to  evaluate  the  vulnerability  and/or  lethality  aspects  of  a  conventional 
weapon  or  conventional  weapon  system.  Live  Fire  Test  and  Evlauation  (LFT&E)  is 
a  statutory  requirement  (Title  10  U.S.C.  2366)  for  covered  systems,  major  munitions 
programs,  missile  programs,  or  product  improvements  to  covered  systems,  major 
munitions  programs,  or  missile  programs  before  they  can  proceed  beyond  Low-Rate 
Initial  Production  (DAU  2005). 

Low-Rate  Initial  Production 

The  minimum  number  of  systems  (other  than  ships  and  satellites)  to  provide  production- 
representative  articles  for  IOT&E,  to  establish  an  initial  production  base,  and  to  permit 
an  orderly  increase  in  the  production  rate  sufficient  to  lead  to  Full- Rate  Production  upon 
successful  completion  of  IOT&E  (DAU  2005). 

Maintainability 

The  ability  of  an  item  to  be  retained  in,  or  restored  to,  a  specified  condition  when 
maintenance  is  performed  by  personnel  having  specified  skill  levels,  using  prescribed 
procedures  and  resources,  at  each  prescribed  level  of  maintenance  and  repair  (DAU  2005). 

Major  System  Deficiency 

A  system  shortfall  that  adversely  affects  the  accomplishment  of  an  operational  or  mission 
essential  capability  and  no  known  work  around  is  available. 

Marine  Officer  in  Charge 

The  Marine  Officer  in  Charge  (MOIC)  is  responsible  for  helping  to  execute  the  test  plan 
and  report  the  test  deviations  to  the  OTPO.  The  MOIC  is  also  responsible  for  helping 
to  coordinate  necessary  resources  required  to  support  tests;  supervising  the  Marines 
conducting  the  events  described  in  Trial  Conduct  and  ensuring  that  Marines  collect  data 
specified  in  Data  Requirements;  ensuring  that  the  Marines  collect  the  data  in  accordance 
with  the  Test  Plan;  maintaining  a  daily  log  that  includes  significant  events  and  incidents 
that  affect  test  conduct,  test  events  completed,  and  personal  observations  of  the  test 
conduct  and  system  functionality;  tracking  the  daily  review,  editing,  and  compliation  of  all 
data  collection  forms  and  electronic  data  collection;  and  reviewing  TIRs  for  accuracy  and 
completeness  and  provide  preliminary  scoring  ofTIRs  for  scoring  conference  members. 

Mean  Time  Between  Failure 

A  basic  measure  of  reliability  for  repairable  items.  The  average  time  during  which  all  parts 
of  the  item  perform  within  their  specified  limits,  during  a  particular  measurement  period 
under  stated  conditions  (DOD  2005). 
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Mean  Time  Between  Maintenance 

A  basic  measure  of  reliability  for  repairable  fielded  systems.  The  average  time  between 
all  system  maintenance  actions.  Maintenance  actions  may  be  for  repair  or  preventive 
purposes  (DOD  2005). 

Mean  Time  To  Repair 

An  estimate  of  the  expected  amount  of  time  (minutes,  hours,  days,  etc.)  to  perform  a 
corrective  maintenance  action.  Requires  a  definition  of  a  corrective  maintenance  action. 
Disregards  the  time  ordering  of  time-to-repair  data. 

Measure 

Provides  the  basis  for  describing  varying  levels  of  performance,  i.e.,  the  dimensions, 
quantity,  or  capacity  of  something  as  ascertained  by  a  quantitative  or  qualitative  value 
(Draft  MOT&E  MOA  2007).  Often  accompanied  by  a  standard. 

Measure  of  Effectivness 

A  Measure  of  Effectiveness  is  designed  to  correspond  to  the  accomplishment  of  mission 
objectives  and  achievement  of  desired  results. 

Measure  of  Suitability 

A  Measure  of  Suitability  measures  an  item’s  ability  to  be  supported  in  its  intended 
operational  environment. 

Measure  of  Survivability 

A  Measure  of  Survivability  is  designed  to  measure  the  degree  to  which  the  system  or  the 
system  operators  are  placed  at  risk  in  an  operational  environment.  It  may  also  measure  the 
degree  to  which  the  system  places  other  systems/operators  at  risk.  For  information  and 
business  systems,  this  is  based  on  Information  Assurance. 

Measure  of  Performance 

A  Measure  of  Performance  measures  a  system’s  performance  expressed  as  speed,  payload, 
range,  time-on-station,  frequency,  or  other  distinctly  quantifiable  performance  features. 

Milestone 

The  point  at  which  a  recommendation  is  made  and  approval  sought  regarding  starting  or 
continuing  an  acquisition  program,  i.e.,  proceeding  to  the  next  phase  (DAU  2005). 

Milestone  Decision  Authority 

Designated  individual  with  overall  responsibility  for  a  program.  The  Milestone  Decision 
Authority  (MDA)  shall  have  the  authority  to  approve  entry  of  an  acquisition  program 
into  the  next  phase  of  the  acquisition  process  and  shall  be  accountable  for  cost,  schedule, 
and  performance  reporting  to  higher  authority  (DODD  2007). 

Mission 

The  high-level  Task,  together  with  the  purpose,  that  clearly  indicates  the  action  to  be 
taken  and  the  reason  therefore.  (CJCSM  2005). 
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Mission-Based  Testing 

A  test  methodology  that  emphasizes  the  evaluation  of  a  system’s  contribution  to  mission 
success  and  the  degree  to  which  it  delivers  specified  and  implied  capabilities,  rather  than 
the  verification  of  specifications.  The  MBT  design  process  is  structured  on  the  following 
basic  elements:  mission  analysis,  system  performance  measures,  operating  conditions,  and 
test  variables. 

Mission  Capability  Level 

Mission  Capability  Level  (MCL)is  used  for  all  systems  being  evaluated  for  OE/OS/ 
OSur.  Determining  MCL  is  not  required  by  law  or  directive,  but  it  provides  a  systematic 
means  of  arriving  at  the  required  conclusions  for  OE/OS/OSur.  A  determination  of 
Mission  Capability  Level  expresses  to  the  decision  maker,  on  a  by-mission  basis,  the  level 
of  performance  that  can  be  expected  of  the  system  for  a  particular  mission. 

Mission  Essential  Functions 

Minimum  functional  capabilities  that  a  system  must  possess  to  be  considered  mission 
capable  (DAU  2005).  Mission  Essential  Functions  (mef)  maybe  derived  from  capability 
documents  or  developed  during  the  charter  with  DC,  CD&I’s  concurrence.  Mefs  are 
usually  expressed  as  an  action  verb  that  describes  a  necessary  capability,  i.e.,  shoot,  move,  etc. 

Model  and  Simulation 

A  model  is  a  physical,  mathematical,  or  otherwise  logical  representation  of  a  system, 
entity,  phenomenon,  or  process.  A  simulation  is  a  method  for  implementing  a  model 
over  time.  Also,  it  can  be  a  technique  for  testing,  analysis,  or  training  in  which  real-world 
systems  are  used  or  where  real-world  and  conceptual  systems  are  reproduced  by  a  model 
(DODD  2007). 

Mutual  Exclusivity 

The  same  objective  should  be  covered  only  once  in  the  evaluation  hierarchy;  no  overlap 
should  occur  between  the  COIs 

Net- Ready  Key  Performance  Parameter 

DOD-directed  Key  Performance  Parameter  that  assesses  the  information  needs, 
information  timelines,  Information  Assurance,  and  net-ready  Attributes  required  for  both 
the  technical  exchange  of  information  and  the  end-to-end  operational  effectiveness  of 
that  exchange  (CJCSI  2005). 

Not  Mission  Capable 

The  system  does  not  improve  on  current  mission  capabilities  for  the  desired  effect  or 
outcome. 

Observation  Plan 

Plan  generated  by  the  test  team  to  observe  DT  events.  States  the  purpose  of  the  event, 
the  members  attending,  examines  attributes  with  thresholds,  and  pulls  evaluation 
questions  from  the  SEP. 
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Observation  Report 

Report  generated  by  the  test  team  upon  return  from  a  DT  event  that  documents 
observations  with  no  analysis. 

Off-the-Shelf 

Procurement  of  existing  systems  or  equipment  without  a  Research,  Development,  Test 
and  Evaluation  program  or  with  minor  development  necessary  to  make  system  suitable 
for  DOD  needs.  May  be  a  commercial  system/equipment  or  one  already  in  the  DOD 
inventory  (DAU  2005). 

Operational  Assessment 

MCOTEA  may  conduct  Operational  Assessment  (OA)  to  demonstrate  selected  system 
performance,  with  user  support  as  required.  An  OA  can  range  from  a  “paper  assessment” 
to  modeling  and  simulation  to  a  physical  operational  test.  The  nature  of  the  OA  is 
described  in  the  TEMP.  An  OA  can  be  conducted  at  any  time,  but  is  normally  done 
during  the  Engineering  and  Manufacturing  Development  phase  of  the  acquisition  cycle 
to  evaluate  selected  Issues,  KPPs,  and  other  system  attributes.  An  OA  typically  focuses 
on  significant  trends  noted  in  developmental  efforts,  programmatic  voids,  areas  of  risk, 
testability  of  capabilities,  and  the  ability  of  the  program  to  support  adequate  operational 
testing.  An  OA  does  not  determine  OE,  OS,  or  OSur. 

Operational  Availability 

The  degree  (expressed  as  a  decimal  between  0  and  1,  or  the  percentage  equivalent)  to 
which  one  can  expect  a  piece  of  equipment  or  weapon  system  to  work  properly  when  it  is 
required;  that  is,  the  percent  of  time  the  equipment  or  weapon  system  is  available  for  use. 
Ao  represents  system  “uptime”  and  considers  the  effect  of  reliability,  maintainability,  and 
mean  logistics  delay  time  (DAU  2005). 

Operational  Deficiency 

Issues  that  impact  the  performance  of  the  system  under  test.  They  tend  tend  to  pertain 
to  interfaces  with  other  systems  or  to  interactions  with  the  operating  forces.  In  some 
cases,  these  deficiencies  may  actually  be  materiel  gaps  in  operational  capability  and  in 
other  cases,  they  may  illuminate  the  need  to  create  or  modify  tactics,  techniques,  and 
procedures. 

Operational  Effectiveness 

Measure  of  the  overall  ability  of  a  system  to  accomplish  a  mission  when  used  by 
representative  personnel  in  the  environment  planned  or  expected  for  operational 
employment  of  the  system  considering  organization,  doctrine,  tactics,  supportability, 
survivability,  vulnerability,  and  threat  (CJSCI  2009). 

Operational  Mission  Failure 

A  Test  Incident  Report  that  is  scored  as  a  failure  during  the  FD/SC  Conference  because 
the  severity  of  the  failure  rendered  the  system  unable  to  complete  a  mission  essential 
function. 

Operational  Mode  Summary/Mission  Profile 

The  Operational  Mode  Summar/Mission  Profile  (OMS/MP)  is  a  mandatory  appendix 
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to  all  Marine  Corps  capability  documents  and  describes  how  a  system  or  training  device 
will  be  used  in  wartime  and/or  peacetime  at  the  time  it  is  fielded  with  focus  on  the 
future.  Information  in  an  OMS/MP  presents  a  structured,  quantitative  picture  of  annual 
equipment  usage. 

Operational  Suitability 

The  degree  to  which  a  system  can  be  placed  and  sustained  satisfactorily  in  field  use  with 
consideration  being  given  to  availability,  compatibility,  transportability,  interoperability, 
reliability,  wartime  usage  rates,  maintainability,  safety,  human  factors,  habitability, 
manpower,  logistics  supportability,  natural  environmental  effects  and  impacts, 
documentation,  and  training  requirements  (CJCSI  2005). 

Operational  Task  Analysis 

MCOTEA  uses  operational  task  analysis  as  the  analytic  backbone  of  the  Evaluation 
Framework.  Task  Analysis  supports  evaluations  by  breaking  down  complex  missions  into 
their  component  Tasks  and  Subtasks.  Operational  Task  Analysis  provides  a  disciplined 
method  for  developing  the  framework  for  evaluation  questions  below  the  level  of  OE, 
OS,  and  OSur.  Operational  Task  Analysis  is  top-down  and  mission-based. 

Operational  Test  and  Evaluation 

The  field  test,  under  realistic  conditions,  of  any  item  (or  key  component)  of  weapons, 
equipment,  or  munitions  for  the  purpose  of  determining  the  effectiveness  and  suitability 
of  the  weapons,  equipment,  or  munitions  for  use  in  combat  by  typical  military  users;  and 
the  evaluation  of  the  results  of  such  tests  (CJSCI  3170.01E). 

Operational  Test  Readiness  Board 

Approximately  90  days  before  NET,  MCOTEA  will  conduct  an  Operational  Test 
Readiness  Board  (OTRB).  Participants  include  the  MOIC,  representatives  from  the  PM, 
ASN(RD8cA)  (for  ACAT  I  and  II  programs),  MCSC  Executive  Commander,  Programs 
and  Chief  Engineer,  and  DC,  CD&LThe  purpose  of  the  OTRB  is  to  determine 
the  readiness  of  a  system,  support  packages,  instrumentation,  test  planning,  and  test 
participants  to  support  the  OT.  It  identifies  any  problems  that  may  affect  the  start  or 
proper  execution  of  the  OT  and  makes  any  necessary  changes  to  test  plans,  resources, 
training,  or  equipment. 

Parameter 

Any  characteristic  of  the  population  that  is  sought  from  a  system  under  test  from  the 
capabilities  documents  and  is  represented  by  the  threshold  and  objective  values  or  actual 
test  data. 

Partially  Mission  Capable 

A  system  that  is  considered  to  be  at  least  as  good  as  the  current  capability,  but  still  falls 
short  of  the  threshold  for  the  desired  effect  or  outcome. 

Pilot  Test 

Used  to  rehearse  the  record  test,  to  evaluate  the  operational  test  methodology,  and  to 
ensure  that  data  collection  forms  and  procedures  are  verified,  that  issues  involving  the 
operation  of  the  test  equipment  are  resolved,  that  data  analysis  procedures  in  the  Test 
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A  scheduling  management  tool  that  contains  projected  operational  test  dates,  a  list  of 
milestone  dates  for  completing  the  Test  Plan,  the  FoS,  and  the  OER,  and  intermediate 
dates  for  all  supporting  activities. 

Preliminary  Design  Review 

A  multi-disciplined  technical  review  to  ensure  that  a  system  is  ready  to  proceed  into 
detailed  design  and  can  meet  stated  performance  requirements  within  cost  (program 
budget),  schedule  (program  schedule),  risk,  and  other  system  constraints  (DAU  2005). 

Program  Manager 

Designated  individual  with  responsibility  for  and  authority  to  accomplish  program 
objectives  for  development,  production,  and  sustainment  to  meet  the  user’s  operational 
needs.  The  PM  shall  be  accountable  for  credible  cost,  schedule,  and  performance  reporting 
to  the  Milestone  Decision  Authority  (DoDD  2007).  Usually  a  colonel  at  MCSC  or  a 
GS-equivalent. 

Quad  Chart 

Briefing  slide  tool  used  in  acquisitions.  The  four  quadrants  of  a  Quad  Chart  show  a  photo 
or  graphic  of  the  system  in  an  operational  setting,  a  brief  description  of  the  operational 
capability  of  the  system,  the  proposed  technical  approach  of  the  tasks  to  be  performed  to 
acquire  or  develop  the  system,  and  a  cost  and  development  schedule. 

Record  Test 

The  portion  of  the  operational  test  that  produces  the  official  data  that  will  be  used  to 
evaluate  the  system.  Usually  follows  a  pilot  test. 

Reliability 

The  ability  of  a  system  and  its  parts  to  perform  its  mission  without  failure,  degradation,  or 
demand  on  the  support  system.  Often  expressed  as  a  probability,  i.e.,  the  probability  that 
the  system  will  perform  its  mission  profile  or  mission  duration  without  a  failure  (DAU 
2005). 

Reliability,  Availability,  and  Maintainability 

Requirement  imposed  on  acquisition  systems  to  ensure  the  following:  systems  are 
operationally  ready  for  use  when  needed,  systems  will  successfully  perform  assigned 
functions,  and  systems  can  be  economically  operated  and  maintained  within  the  scope 
of  logistics  concepts  and  policies.  Reliability,  Availability,  and  Maintainability  (RAM) 
programs  are  applicable  to  systems;  test  measurement  and  diagnostic  equipment;  training 
devices;  and  facilities  developed,  produced,  maintained,  procured,  or  modified  for  use. 

See  also  individual  definitions  for  Reliability,  Availability,  and  Maintainability  (DAU 
2005). 

Rough  Order  of  Magnitude 

Estimate  of  cost  based  on  approximate  cost  models  or  expert  analysis.  Usually  based 
on  top-level  requirements  or  specifications  and  an  overall  prediction  of  work  to  be 
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done  to  satisfy  the  requirements.  Used  primarily  for  financial  planning  purposes  (www. 
maxwideman.com/pmglossary/PMG_R06. htm).  At  MCOTEA  the  OTPO  prepares  a 
program’s  ROM,  which  includes  the  test  concept,  the  schedule  of  OT  phases,  the  cost 
estimate,  and  initial  program  issues.  During  the  program  definition  phase  of  the  test 
process,  the  OTPO  provides  the  ROM  to  the  Branch  Head,  and  then  to  the  system 
Project  Officer  and  the  DC,  CD&I  Materiel  Capabilities  Officer. 

Screening  Criteria 

A  binding  constraint  on  the  system  evaluation,  which  can  reduce  the  number  of  Issues  to 
only  those  essential  for  determining  worth  or  value. 

Statement  of  Need 

A  Statement  of  Need  (SON)  is  prepared  by  the  DC,  CD&I  action  officer  in  lieu  of 
a  JCIDS  capabilities  document  (ICD,  CDD,  or  CPD)  to  define  the  attributes  for  a 
capability  that  because  of  its  projected  cost,  will  meet  the  requirements  for  an  Abbreviated 
Acquisition  Program,  or  due  to  an  unusual  and  compelling  urgency  (i.e.,  war),  when 
traditional  JCIDS  documentation  would  be  unresponsive  to  the  current  operational  need 
(CDD  Handbook). 

Subtasks 

Tasks  are  subdivided  into  lower  level  “Subtasks. ’’These  supporting  Subtasks  constitute  the 
discrete  actions  that  must  occur  to  accomplish  the  task.  Some  Subtasks  may  be  associated 
with  more  than  one  Task. 

System  Assessments 

System  Assessments  pertain  to  programs  being  tested  or  examined  at  less  than  full  IOT, 
such  as  Quick  Reaction  Assessments  (QRA),  AAPs,  ACAT  IV(M)  programs,  and  non- 
Programs  of  Record.  MCOTEA  uses  this  type  of  assessment  to  help  the  decision  maker 
determine  a  system’s  capabilities  and  limitations. 

System  Evaluation  Plan/System  Assessment  Plan 

The  data  analysis  plan  that  is  the  roadmap  for  a  system  evaluation  or  system  assessment. 

A  System  Evaluation  Plan  (SEP)  is  used  for  Intermediate  Assessments,  Operational 
Assessments,  and  integrated  and  Operational  Tests.  A  System  Assessment  Plan  (SAP)  is 
used  for  System  Assessments. 

System  Threat  Assessment 

Describes  the  threat  to  be  countered  and  the  projected  threat  environment.  The  threat 
information  must  be  validated  by  the  Defense  Intelligence  Agency  for  programs  reviewed 
by  the  Defense  Acquisition  Board  (DAU  2005).  MCOTEA  should  correspond  with  the 
DC,  CD&I  action  officer  for  specific  information  concerning  a  program’s  STA. 

System  Under  Test 

The  test  article  undergoing  the  IOT&E.The  solution  undergoing  the  acquisition  process. 

Tactical  and  Exercise  Employment  Plan 

Automated  software  system  designed  to  support  planning  and  execution  and  to  provide 
visibility  of  training,  exercises,  and  deployment  activities  throughout  the  FMF. 
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The  system  allows  FMF  commanders  (battalion/squadron)  and  higher  level  staffs  to 
plan  and  project  training,  exercises,  and  employment  activities  to  ensure  the  prudent 
expenditures  of  resources  (personnel,  equipment,  and  money)  while  still  fulfilling  mission 
requirements  (DON  1996). 

Tasks 

Founded  in  the  capabilities  the  system  is  intended  to  address.  Tasks  are  used  to  frame 
additional  questions  at  a  lower  level  than  the  COIs. 

Test  and  Evaluation  Master  Plan 

Documents  the  overall  structure  and  objectives  of  the  Test  and  Evaluation  program.  It 
provides  a  framework  within  which  to  generate  detailed  test  and  evaluation  plans  and 
documents  schedule  and  resource  implications  associated  with  the  test  and  evaluation 
program.  The  Test  and  Evaluation  Master  Plan  (TEMP)  identifies  the  necessary  DT&E, 
IOT&E,  and  LFT&E  activities  required  to  answer  evaluation  questions  and  determine 
the  satisfaction  of  thesholds  in  the  capabilities  documentation.  It  relates  program 
schedule,  test  management  strategy  and  structure,  and  required  resources  to  the  following: 
Critical  Operational  Issues,  Critical  Technical  Parameters,  objectives  and  thresholds 
documented  in  the  Capability  Development  Document,  evaluation  criteria,  and  milestone 
decision  points  (DAU  2005). 

Test  Data  Report 

Report  generated  by  the  test  team  upon  returning  from  System  Assessments  (System 
Assessment  Test  Report),  Operational  Assessments  (Early  Operational  Assessment  Test 
Report  or  Operational  Assessment  Test  Report),  and  tests  that  contains  raw  data  results 
with  no  analysis. 

Test  Limitations 

Shortfall  in  OT  depth  or  breadth  that  may  affect  the  resolution  of  a  test  Issue. 

Test  Incident 

Any  unintended  occurrence  that  takes  place  during  test. 

Test  Incident  Report 

Form  used  to  record  unintended  occurrences  during  system  test.  Test  Incident  Reports 
that  affect  RAM  are  scored  at  the  Failure  Definition/Scoring  Criteria  Scoring 
Conference  and  are  used  to  resolve  RAM  characteristics. 

Test  Plan 

Plan  written  by  the  test  team  for  System  Assessments,  Operational  Assessments,  and 
Operational  Tests  for  test  execution.  It  includes  a  schedule,  test  team  organization, 
MCOTEA’s  plan  for  the  data  obtained,  data  requirements,  data  collection  methods,  trial 
sequence,  trial  conduct,  and  logistical  information. 

Threshold 

A  minimum  acceptable  operational  value  below  which  the  utility  of  the  system  becomes 
questionable  (DAU  2005). 
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Verification,  Validation,  and  Accreditation 

Verification:  The  process  of  determining  that  an  M&S  implementation  and  its  associated 
data  accurately  represent  the  developer’s  conceptual  description  and  specifications. 

Validation:  The  process  of  determining  the  degree  to  which  an  M&S  and  its  associated 
data  are  an  accurate  representation  of  the  real  world  from  the  perspective  of  the  intended 
use  of  the  M&S. 

Accreditation:  The  official  determination  that  an  M&S  application  and  its  associated 
data  are  acceptable  for  use  for  a  specific  purpose. 

Vulnerability  Testing 

Vulnerabilty  LFT&E  focuses  most  specifically  on  the  system’s  reponse  once  a  threat 
affects  the  system,  i.e.,  penetration  and  kill. 


G-16 


Acronyms 


Acronyms 


AAP 

ACAT 

ACMC 

ACOR 

ADM 

ALO 

ANOVA 

A 

0 

AoA 

ARL 

ASN(RDA) 

Abbreviated  Acquisition  Program 

Acquisition  Category 

Assistant  Commandment  of  the  Marine  Corps 
Assistant  Contracting  Officer’s  Representative 
Acquisition  Decision  Memorandum 

Air  Liaison  Officer 

Analysis  of  Variance 

Operational  Availability 

Analysis  of  Alternatives 

Army  Research  Laboratory 

Assistant  Secretary  of  the  Navy  (Research,  De¬ 
velopment,  and  Acquisition) 

BAD 

BDAR 

BH&T 

C2 

C4ISR 

Behind  Armor  Debris 

Battle  Damage  Assessment  and  Repair 

Ballistic  Hull  &  Turret 

Command  and  Control 

Command,  Control,  Communications,  Comput¬ 
ers,  Intelligence  and  Reconnaissance 

CAC 

CAE 

CBRN 

CBT 

CD&I 

CDD 

CDT 

COE 

COI 

COMOPTEVFOR  or  COTF 

Common  Access  Card 
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